############################################################################ ##########robots.txt file by Tito Lukic - last revised on Tuesday, 19.01.2010. ############################################################################ ##########http://www.robotstxt.org/ ##########http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html ##########http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360 ##########http://user-agents.org/index.shtml ##########http://en.wikipedia.org/wiki/Robots.txt ##########http://www.javascriptkit.com/howto/robots.shtml ##########etc ... ############################################################################ User-agent: Googlebot ####################folders Disallow: /affiliates/ Disallow: /classifieds/ Disallow: /cgi-bin/community/ Disallow: /directory/Detailed/printerfriendly/ Disallow: /directory/Detailed/remoterating/ Disallow: /forms/ Disallow: /links/ Disallow: /redirect/ Disallow: /store/ Disallow: /web-optimizer/ ####################pages Disallow: /cgi-bin/directory/page.cgi?page=localsearch Disallow: /company.shtml Disallow: /copyright.shtml Disallow: /directory/addalink.html Disallow: /directory/usefullinks.html Disallow: /disclaimer.shtml Disallow: /privacy.shtml Disallow: /search.shtml Disallow: /TOS.shtml ####################forum issues Disallow: /*guest= Disallow: /*username= ####################cgi files Disallow: /*bookmark.cgi Disallow: /*contact_form.cgi Disallow: /*listings.cgi Disallow: /*modify.cgi Disallow: /*rate.cgi Disallow: /*review.cgi Disallow: /*subscribe.cgi Disallow: /*user.cgi ####################spammers Disallow: /*adobt Disallow: /*adult Disallow: /*anal Disallow: /*API Disallow: /*cock Disallow: /*erotic Disallow: /*fuck Disallow: /*gangbang Disallow: /*masturbation Disallow: /*naked Disallow: /*nude Disallow: /*orgasm Disallow: /*pedo Disallow: /*porn Disallow: /*pussies Disallow: /*pussy Disallow: /*sex Disallow: /*spanking Disallow: /*xanax Disallow: /*xxx ####################unreadable or mistakes Disallow: /*£ Disallow: /*¥ Disallow: /*¤ Disallow: /*¶ Disallow: /*€™ Disallow: /*¢ Disallow: /*¬ Disallow: /*¾ Disallow: /*± Disallow: /*† Disallow: /*© Disallow: /*ƒ Disallow: /*Ø Disallow: /*§ Disallow: User-agent: Mediapartners-Google Disallow: User-agent: Slurp Disallow: User-agent: msnbot Disallow: User-agent: Teoma Disallow: User-agent: Jeeves Disallow: User-agent: * Disallow: / ############################################################################ ##########End of the file. ############################################################################