how can i protect google or other search engine to search my whmcs ?

[rob2]

Hello,

 

when i google my whmcs's domain,

 

i can not some records directly point to my whmcs,

 

how can i protect google or other search engine to search my whmcs ?

 

because i worry they may find out my internal data some day.

 

 

 

thank you

[disgruntled]

robots.txt Google it.

[bear]

Not all spiders honor robots.txt. Google *usually* does, but there are lots that do not.

[disgruntled]

Not all spiders honor robots.txt. Google *usually* does, but there are lots that do not.

 

Yes i know, but the bad bots are really easy to deal with,

 

Add into your robots.txt a deny on directory, in this directory you have a script that adds ipblocking to the firewall.

 

Good bots will honour your robots.txt bad bots will not and will thus be ipbanned from your server. i have used this model for many years its certainly good for keeping the bad bots at bay. you need to log the details though, occasionally a bot you want will enter into the directory and thus be banned, so you essentially lose that bot until you remove the ban.

[rob2]

Yes i know, but the bad bots are really easy to deal with,

 

Add into your robots.txt a deny on directory, in this directory you have a script that adds ipblocking to the firewall.

 

Good bots will honour your robots.txt bad bots will not and will thus be ipbanned from your server. i have used this model for many years its certainly good for keeping the bad bots at bay. you need to log the details though, occasionally a bot you want will enter into the directory and thus be banned, so you essentially lose that bot until you remove the ban.

 

can you share more detail with me about how do you do it ? thank you