Remove robots.txt

Google’s documentation says:

> robots.txt is not a mechanism for keeping a web page out of Google. To
> keep a web page out of Google, you should use noindex directives

We’ve implemented a noindex directive now, so we don’t need to serve
robots.txt any more.
This commit is contained in:
Chris Hill-Scott
2020-05-26 17:47:43 +01:00
parent f902205ef3
commit f12f0fae87
3 changed files with 2 additions and 27 deletions

View File

@@ -36,17 +36,6 @@ def index():
)
@main.route('/robots.txt')
def robots():
return (
'User-agent: *\n'
'Disallow: /sign-in\n'
'Disallow: /support\n'
'Disallow: /support/\n'
'Disallow: /register\n'
), 200, {'Content-Type': 'text/plain'}
@main.route('/error/<int:status_code>')
def error(status_code):
if status_code >= 500: