We would first like to apologise for the past 4-6 weeks of Marapets Maintenance issues. Normally we would keep you all updated but this time we had to keep quiet as we were running on much less server power than normal and making this public, somebody could have used this information to crash us!
Half of the problem was that we purchased all of our current infrastructure at once, 5 years ago. We had 24 different 15KRPM SSCI hard drives and they were failing at one per week!
We had ordered many spares and were replacing them as they failed. The issue was that as one disk failed, more load was transferred to others, making them fail even quicker!
This sometimes resulted in random crashes that took a while to recover.. several crashes in the past month!
We decided to replace all 24 disks with faster SSD drives now which cost a lot to purchase all at once and also took up a lot of time to replace, as we replaced 4 at a time without the site going down.
When we set up our infrastructure 5-6 years ago, we used high performance InfiniBand for our network. This is amazingly fast and worked well for us for many of those years. As the years went by, less and less software and updates were compatible with IB and 10 Gigabit Ethernet (10GbE) got much cheaper.
Last month we had 24 hours of maintenance where we switched from InfiniBand to 10GbE. This was a complete success and many players almost instantly noticed how much faster Marapets was running... until the timeouts and appeared! This was mainly down to our older hard drives being replaced and rebuilt in the background.
All of this maintenance has now been completed
Everything *should* now be back to normal and running faster than ever. All of our time has been spent on this maintenance and the next couple of days will be spent fixing any of other errors that we have missed.
We would like to thank you for your patience and normal news updates should now be back!
*fingers and toes crossed*
