[GUIDE] Catch unidentified Bots - Fix

[Stoker]

Catch unidentified Bots - Fix

We want to avoid unidentified Bots crawling our board.
Not only will the cause excessive load and traffic. They will also have access to the SIDs in the ending of some of the URLs and lots of other stuff they dont need to access.

With this simple Bot fix we will handle ALL bots, crawlers etc who has the "bot" in the agens string.
All you have to do is add a new bot in ACP.

The Bot name doesnt matter. I have called mine: Unidentified [Bots]
Agent match must be: bot

Now all unidentified bots with the agent match "bot" will only have bot access. Like any other bot.

Unidentified-[Bots].png

Credits: https://forum.webseodesigners.com/web-d ... t5456.html


Risk of False Positives:
Matching on "bot" is a very broad filter. It could accidentally treat non-malicious or even legitimate tools as bots, especially if:

• A browser plugin adds "bot" to the agent string
• A custom app with "bot" in the name is used by a real person

[Steve]

It would be better to match the key word and then submit the user agent for review in a log some where. "bot" is included in words like "bottom"

[1ubz32cb9]

It would be better to match the key word and then submit the user agent for review in a log some where. "bot" is included in words like "bottom"

Bottom is not present in any known User Agent,see https://gist.github.com/pzb/b4b6f57144aea7827ae4

The ideal for identifying each individual bot would be adding detected bots to the phpBB bot list with the whole word that contains the string "bot", but that would increase forum load which would defeat the purpose of this fix. When I wanted to find individual bots and identify them, I have grepped web server logs with the last visit time and again with "bot" like this:

Code: Select all

grep 2025:10:15:12 LOG_FILE | grep -i bot

There are only some 1-2 new bots/spiders/crawlers per week at the moment in my forum. However, the rate is growing rather rapidly.