Slow Forum?

[Martijn Meerts]

It seems to be really random when it happens, and then for how long it remains slow. I have various system logs and performance logs, but there's nothing sticking out as being an obvious cause of the issue.

 

I haven't gotten around to setting up the new server instance yet, and I'm not 100% sure that will actually fix things. But either way, we do need the update at some point soon-ish, since the forum software is also heading towards a new major release with new minimum requirements that we likely don't meet at the moment.

 

[Kingmeow]

Seems to be back to normal for the past few days.  Just another data point.

[cteno4]

It appears to be something that spreads. This time first report was from canada, but still working fine hee in the us, but then soon punky here, then reports from all over. Then after a couple of days it all goes poof. This seems to happen maybe every 3-6 months and each time Martijn has not found any indications in the logs of something gumming up ther works so it seems to be outside the server and in the aws connections to the world. Fun in the world of IT! 

 

Jeff

[Martijn Meerts]

Due to the situation in the world, there's also been a lot more attacks on various of the big hosting companies. While those attacks aren't directed at the forum specifically, they will cause a lot of additional traffic on the data centers itself, which in turn can cause the forum to slow down.

 

That said, I do believe it's not just the hosting provider in this case, but I haven't been able to find any cause (yet). I do still plan on spinning up a new server instance with updated software, I just need to make some time for that at some point.

[cteno4]

The very slow page loading seems to be back again here in DC. Started earlier today. Takes a minute so for a page to load

 

jeff

[Martijn Meerts]

Definitely slower than usual for me as well at the moment. Not 1 minute for a page to load, but certainly not very fast either.

 

 

Update: Just after posting, it seems the forum stopped responding completely. I just rebooted the instance, and it seems to be working again now. In the coming days I will look into setting up a new server instance with updated software. Once I'm ready to move the forum over to that new instance, I'll make a new post about that.

 

I can't really give an indication of when that'll be just yet, I have a lot of things going on at the moment, many of which aren't positive. But I'll try to start looking into it some time in the coming days.

 

[Madsing]

Oh, yes, the reboot definitely repaired something. Now it’s working again. Thank you Martijn!

[Martijn Meerts]

Hi all!

 

As you are no doubt aware, to forum has slowed down again. After doing a little research, I think we may have finally found a reason for this. It won't be a quick fix, and I'm not sure when we'll be able to work on it, but at least we know where to look now.

 

The first step will still be updating the server software and the forum software. That in itself might alleviate some of the issues. Due to a severe family health scare, I can't really say when I can sit down and get that done however.

[Kingmeow]

Martijn, I haven't really noticed much of a slow down on my end (US east coast).  But, don't worry about us and take care of your family first.  This is only a hobby despite some of us who are ultra passionate about it.  You only have one family.  My best wishes for a positive outcome.

 

Edit: Oops!  I spoke too soon.  Now I'm experiencing the slow down but not too bad.

Edited May 16 by Kingmeow

[RS18U]

3 hours ago, Kingmeow said:

Martijn, ... don't worry about us and take care of your family first.  This is only a hobby despite some of us who are ultra passionate about it.  You only have one family.

What @Kingmeow said.  @Martijn Meerts you need to take care of yourself and family.  We can get by.  Actually more time modeling than posting 😀

[cteno4]

Yes Martjin, dont worry about the server, it works. Family first.

 

jeff

[Martijn Meerts]

Hey all!

 

First of all, thanks for the kind words, it's appreciated.

 

 

I've done a little more research in the cause of the slow downs, and while I know why the forum slows done, I'm still not sure what the underlying cause of it is. We're running the forum on AWS. We have 1 server for the database and 1 server for the website. The database has been doing fine, but the webserver has been seeing several CPU spikes over the past weeks. Basically, in order to keep cost down, we're using AWS's general purpose instance types. These are relatively cheap for still very decent performance. The way they keep it cheap, is that you build up "CPU credits" when you're not taxing the CPU very much. This is usually the case most of time on a webserver with not a great deal of traffic. When the CPU load goes above a certain threshold, CPU credits will get used for that additional load. Once the CPU credits are used up, the server basically gets limited CPU power. And that's exactly what's been happening the past few weeks.

 

So, the question is, what is causing those CPU spikes? I've noticed that the spikes coincide with a large increase in network traffic. So there is a fair chance some new bots/spiders/scrapers have been introduced which we aren't blocking yet. Another possibility is that the forum is running some background tasks, but while that can cause CPU spikes, it shouldn't cause additional (external) network traffic.

 

So, what I want to do somewhere in the coming days/weeks(/months?) is:

1. Check / update our bots/spiders/scrapers blacklist, and update it if needed
2. Look into some possibilities on the server side to improve performance / decrease network traffic
3. Regardless of whether or not that fixes anything, set up a new server instance with an up-to-date OS, and migrate everything to that instance
4. Update the forum software to the latest 4.x version
5. Should that prove insufficient, look into spreading the load over multiple server instances, if the forum software supports that. This would mean increased cost.
6. Worst case scenario, look into the cloud hosting that Invision Community offers. This would also mean increased cost, but also less flexibility