Software
/
rosa-build
mirror of https://github.com/OpenMandrivaSoftware/rosa-build.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

disable crawler bots

This commit is contained in:
Alexander Khryukin 2016-03-17 15:09:10 +03:00
parent b46a1ab4d5
commit ad77e882af
1 changed files with 2 additions and 18 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
app/views/sitemap/robots.erb
View File

@ -1,22 +1,6 @@
# robots.txt for <%= root_url %> # robots.txt for <%= root_url %>
User-agent: * User-agent: *
Disallow: /*/*/tree/* Disallow: /
Disallow: /*/*/blob/*
Disallow: /*/*/wiki/*/*
Disallow: /*/*/issues?*
Disallow: /*/*/commits/*
Disallow: /*/blame/
Disallow: /*/raw/
Disallow: /*.git
Disallow: /*.git/
Disallow: /search
Disallow: /users/sign_up
Disallow: /users/sign_in
Disallow: /users/password/new
Disallow: /users/register_requests/new
Disallow: /invite.html
Disallow: /thanks.html
Allow: /
Host: <%= root_url.gsub(/\/$/, '') %> Host: <%= root_url.gsub(/\/$/, '') %>
Sitemap: <%= sitemap_url %> Sitemap: <%= sitemap_url %>