Forum Status

[Admin]

The forum should be back to normal... and the cause of the problem was absolutely not on my bingo card.

There were database and PHP issues from heavy use; I basically needed to clear the site out and restore it. Did a bunch of updates, got everything pristine, and it worked awesome... for a bit. Then it went right back to the same issues.

Crawling through the logs, I discovered that the forum site was being absolutely creamed by an AI bot, feeding its learning model. What really shocked me is how many AI bots were hitting us daily, but most of them were no worse than a search engine spider. This one bot, however, was hitting us from over a dozen IP addresses at once and was unrelenting, to the point of being a Denial of Service attack. I'd reset everything and it would work for a couple of minutes, then it would be all over us again. And of course it started getting the same timeouts you were seeing so it would keep repeating its requests.

So the worst of those bots now just get a 403 Forbidden error (many sites seem to like to use the 418 I'm A Teapot error, which I always thought was a joke but it is actually in the spec) when they hit us, and I'm going to check the logs on the game sites to make sure they aren't hitting there as well. The forum server is a much smaller server than the game server, so it's easier to overwhelm.

Thank you so much for your patience during this ordeal. I apologize for not solving it sooner; the evidence was pointing to issues with the underlying server so I wasn't looking for a DoS attack (intentional or not) when it should have been the first thing I checked for.

If you do continue to see forum issues, please continue to send me tickets. Any forum performance issues in the next couple of weeks should be marked as Urgent. Thank you again!

Chris