robots.txt

[mushmoosh]

Any recommendations on how to most effectively configure a robots.txt file for a WHMCS installation?

 

Any directories I should certainly block?

 

Any example robots.txt files I can check out?

 

Thank you!!

[sparky]

This is what I use

User-agent: *

Disallow: /

 

User-agent: Googlebot

Allow: /

Disallow: /cgi-bin/

Disallow: /downloads/

Disallow: /images/

Disallow: /includes/

Disallow: /lang/

Disallow: /modules/

Disallow: /pipe/

Disallow: /status/

Disallow: /templates/

Disallow: /templates/*

Disallow: /templates_c/

 

User-agent: Yahoo-slurp

Allow: /

Disallow: /cgi-bin/

Disallow: /downloads/

Disallow: /images/

Disallow: /includes/

Disallow: /lang/

Disallow: /modules/

Disallow: /pipe/

Disallow: /status/

Disallow: /templates/

Disallow: /templates/*

Disallow: /templates_c/

 

User-agent: Msnbot

Allow: /

Disallow: /cgi-bin/

Disallow: /downloads/

Disallow: /images/

Disallow: /includes/

Disallow: /lang/

Disallow: /modules/

Disallow: /pipe/

Disallow: /status/

Disallow: /templates/

Disallow: /templates/*

Disallow: /templates_c/

[mushmoosh]

You rock!! Thank you so much for your help!

[mushmoosh]

Perhaps this might be a silly question, but I am still learning the details around configuring a robots.txt properly.

 

If WHMCS is hosted on a subdirectory such as /support/, then the robots.txt file should still reside in the root direct; however, each item to disallow should be specified in this manner:

 

Disallow: /support/templates/*

Disallow: /support/tempatels_c/

Disallow: /support/pipe/

 

Or, should the robots.txt be placed in the WHMCS directory named /support/?

[sparky]

yes thats correct... as you have it there (in the root directory)