Server downtime 31 August 2019

[Martijn Meerts]

Hi!

 

As some of you will have undoubtedly noticed, the forum was down for quite some time.

 

The issue was a power outage at the data center. The backup generators did kick in, but they too started going down again for unknown reasons. It took Amazon quite a while to recover all the server instances, but it seems they managed in the end.

 

While this kind of downtime is something we can protect against by adding a server to a 2nd availability zone, it would increase monthly cost quite a bit. We'd also need a load balancer to be set up in front of the 2 servers that makes sure traffic goes where it should. This would probably increase the cost of running the forum by a factor of 3. And while it would be nice to have the forum never be down, I don't believe it's worth the additional cost. Especially since this kind of major outage is very rare for Amazon.

 

Anyway, we seem to be up and running again, if something seems off, please let us know 🙂

 

[cteno4]

Martijn,

 

thanks for the info! No we can easily go without for a little while, best to keep it simple and cheap. Overall amazon seems to have done well except for this bump. But hopefully they will learn from the outage so as not to repeat.

 

jeff

[EdF]

This might be the first time I've seen just 1 as fail, as designed, usually it cascades one way or another and the whole region takes a hit.  But I agree, keep it simple and cheap, a few hours, or even a day down is fine.  Is there a way for us to support the forum in an ongoing way, like patreon.

[cteno4]

We do a donation drive once and a while, it’s been a while, I’ll check with Martijn now the funds are.

 

thanks

 

jeff

[Sheffie]

I don’t think you need to spend more. Reliability has been fine. 

[katoftw]

Agreed. Not worth spending more for a once a year three hour outage.

[Martijn Meerts]

12 hours ago, EdF said:

This might be the first time I've seen just 1 as fail, as designed, usually it cascades one way or another and the whole region takes a hit.  But I agree, keep it simple and cheap, a few hours, or even a day down is fine.  Is there a way for us to support the forum in an ongoing way, like patreon.

 

Yeah, Amazon has been very reliable, and this outage where an entire data center is affected I've only seen once before in the 15 or so years I've been using Amazon's web services.

 

As for support, there's a thread about donations that includes a PayPal donations button, which should still work. We try not to really ask for donations, or link to that thread unless we're low on funds. At the moment, the software license has been recently paid for another 6 months, the servers are paid for on a monthly basis for now. Once I set up the new server (so we can more easily use https and more easily upgrade server software and such), I'll pay up front for a year which will drive cost down.

 

I think we currently still have around 150 USD in our donations account, so we should be fine for a while, especially since I tend to pay about half myself, and get the other half from donations. Donations are always welcome though, and if we start getting too many donations (PayPal has limits to how much donations an account can receive unless it's a registered non-profit), we'll find a way to give that money back through for example a small project party where participants do something like super-detailing a building, and then the winner(s) get some small prices or something.