WHMCS Looping with Web Crawlers

[tkalfaoglu]

As you probably know, the template SIX had some issues with looping with the web crawlers... A typical URL looked like this:

We got tens of thousands of requests like this both from Yandex and Google..

66.249.66.54 - - [14/Nov/2022:13:26:12 +0300] "GET /index.php?rp=%2Fknowledgebase%2Ftag%2Falan-ad%C4%B1language%3Dturkishlanguage%3Dfarsilanguage%3Dportuguese-brlanguage%3Dportuguese-brlanguage%3Dcatalanlanguage%3Dcatalanlanguage%3Dukranianlanguage%3Dswedishlanguage%3Denglishlanguage%3Denglishlanguage%3Dmacedonianlanguage%3Dczechlanguage%3Dromanianlanguage%3Dnorwegianlanguage%3Dczechlanguage%3Drussianlanguage=russian HTTP/1.0" 200 7026 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.5304.110 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

After mentioning it to the support, they said this is fixed in the 8.6.1 release.

However I also wanted them to return 404 if multiple languages are detected in the URL.. That, they refuse to.. So, the thousands of URL's in the web crawlers will stay indefinitely and we will keep getting requests for updates to those URL's forever  unless WHMCS makes this code change.

Am I not thinking this correctly?

 

Edited December 7, 2022 by tkalfaoglu