james/fembooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

slow down crawlers to 1/3 hit/s; nginx will 503 if more than 1/2 hit/s

Browse Source
This commit is contained in:
Shish 2012-06-05 13:12:58 +01:00
parent 8de1bd0c56
commit 16f557954d
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
lib/static/robots.txt
View File

@ -6,3 +6,6 @@ Disallow: /comment/
# to the same images that the index shows # to the same images that the index shows
Disallow: /post/next/ Disallow: /post/next/
Disallow: /post/prev/ Disallow: /post/prev/
# Site is rate limited to 1 request / sec,
# returns 503 for more than that
Crawl-delay: 3