What are all these bots doing here?

[Claire]

You might have seen in the "Who is online?" statistic at the bottom of the campus page usernames that end with [Bot] like Google [Bot] or Bing [Bot] and wonder now: What are these bots? Is the Academy cooperating with Google? Are these bots tracking me? And what exactly are they doing here? Let me answer these question.

First, no these bots do not track you, nor am I cooperating with any of the firms associated with these bots. These bots are on basically every site you visit. They navigate the internet on their own and are not summoned here by me. I have no connection to them at all. The difference to other sites is: the phpBB forum software shows you that a bot is present if it recognizes the bot. In general, the forum is in no cooperation with other sites that would track you. For example, we are NOT using Google analytics. If you look at the cookies that your browser has saved for the forum, you will see that they all come from the forum software itself, none are from third parties.

Now that we got that out of the way: What are these bots?


Crawlers: Google, Bing, DuckDuckGo, Yandex

These bots are the most important ones for the forum. They visit sites on the forum and then transmit what they see to their parent company. That is called crawling. Google or Bing for example essentially take a snapshot from the site they visit, transmit that data to their servers and create an index for the site. Based on that snapshot, the search engine you use decides what to show you, matching it with the words you searched for. If these bots weren't here, nobody would be able to find us via Google or other search Engines.

Google [Bot] is by far the most important of these bots. It creates the index for Google. Then there is Bing [Bot]. It creates the index for Microsoft's search engine bing. But that index is also used by DuckDuckGo, MSN, Yahoo and Ecosia. If Bing [Bot] wasn't here, nobody would be able to find us via these Google alternatives.

Yandex [Bot] also creates a search index but for the Russian search engine Yandex. Overall, outside of Russia Yandex is not that important.

We are primarily found by Google, then DuckDuckGo, then Bing. Everything else is negligible.


SEO bots: Ahrefs, Semrush, Moz, Barkrowler, Majestic-12

These bots do essentially the same thing as the bots I mentioned before, taking snapshots of the sites they visit. But the data they collect is not used to create an index for a search engine. They collect data for firms that specialize in SEO, Search Engine Optimization. They essentially analyze the internet and try to figure out which sites get found how often and for what and try to help other sites optimize their content such that they get found more often in Google searches (and in other engines). They do for example look which sites are linking to other sites and calculate what they call "domain authority", a measure for how important a site is for the content it provides. The more other sites with a high domain authority link to us, the more do they estimate our own domain authority to be high as well. That they are here is a sign that the forum in general is perceived as relevant. They provide no direct value for us other than that they offer a few freely available statistics that you can look up if you want to.


AI related bots: Amazon, Meta, Open AI, Claude and others I suppose

These bots scout the internet either to fetch data that is fed into LLMs, Large Language Models, like chatGPT. Or they try to improve other AI related services like Amazon's Alexa. So if you ever ask Alexa where you can find the best gang rape stories and it answers Ravishment Academy, then it does so because Amazon [Bot] was here (And for clarity: This was a joke. I suppose that Alexa's content filter would make it refuse to answer the question.


Preview bots: Discord, Facebook

Whenever someone posts a link on Discord or Facebook for example, it shows you a preview of the site. Where does that preview come from? A bot visits the linked site, briefly looks at it and creates that preview for you. That is all they do.


I think these are the major categories of bots that I can think of. Over the last few days I "added" a few bots to the forum. And when I say "added" I mean that I made it so that phpBB, the forum software, recognizes them. They were here before, but the forum treated them as normal human guests. That means, whenever they looked at a topic, they inflated the view count artificially. The bots that phpBB recognizes don't do that. The bots that I added to those that phpBB recognizes by default are: Yandex, Claude, Moz, Facebook, Discord, Meta, Barkrowler. I might add a few more. I know that there are some crawlers for chinese search engines that occasionally visit the forum. I don't want these to inflate the view count either.

But yeah, that's it I guess. In case you got questions, feel free to ask. But please note that I'm not an expert on what these bots do. This here is just my summary of my surface level understanding.

[Vela Nanashi]

We also have a not actually a bot as well, but that one only shows up when called for in chat, and does not actually see or log or remember anything, it just currently does the rolling, Perverted Mimic[Bot]

In truth it is just some code I added to the chat program to look for commands I have written and then it replies to those as appropriate.