[DevBlog] Blackout and back online

Hello everybody, I’m quite happy to see the services back online and people returning to the plattaform, so far the website is running like a perfect 10 with our new provider, we haven’t had the need for a Day One Patch but we will be heavily monitoring until the 4 years anniversary day of the site around August 10th, in the meanwhile we felt the need to give you a proper backstory of what happened.

SO WHAT HAPPENED?

During the early morning of July 29th, our old privoder Scaleway performed an update to all their database services (the link between PHP and the databases) and doing so the database service went full blackout, at this point all the information we had was that the services were down for maintenance (and during the blackout the memberships were still being charged by our payment provider).
By July 31st, they dropped the hammer telling that a massive blackout corrupted a massive range of databases and we were left in the middle of it. Then we entered full on damage control cancelling all memberships to avoid further issues and starting the hosting recovery procedure of rescuing the website folders (neither the database or the backup was available).

It was on August 2nd that we got a notification that SOME parts of our database were recovered, we rescued the post list (4 years of posts and uploads listed), some of the settings of the site, categories, taxonomy, page system and some paypal validations, unfortunally we did rescued the users list but not the members list (we had the mails and usernames but we didnt have the membership roles or payment dates), we lost a most of the payment receipts (our paypal platform is up to date until the last minute but the database recovery had the last weeks corrupted) and all the membership data was pointing to an empty domain and without any reference so even if we used that data it would have been useless.

By that date the new website was running on a brand new build (still empty) but fully configured with a new set of settings and fresh tools (including those that I desgined myself, specially most of the membership system) and once we got a recovery file we migrated the content databases into the new site and replicated some of the most specific settings that were missing, by August 3rd the wesite was fully operational but fully disbaled to work on ironing the details like proper banners and making some desitions such as removing the $20.00 and $2.00 memberships (those 4 tiers have been there since way before the site itself) becuase the higher one wasn’t offering any benefit over the Gold Tier and the lower one was a no-income point due to fees and taxes, once the visual and new conditions were implemented and validated we take an extra day to verify completly the website before launching it.

BETTER THAN BEFORE

With out new provider the site now have some new and better tools for both safety and performance, for starters the site now have a proper mail delivery service that can be tracked and validated properly, this means less mails into SPAM and any user-error situation (like for example a membership with a wrong email address) can be fixed immediatly and properly reducing the number of ghost memberships. We also now have better security with a full on service to avoid directory leaking (a very big issue on the previous hosting) and better bandwidtch to avoid connection errors during concurrent access.

The website itself now have better security too, working on a subdomain level to keep the blog safe and clean, better SSL and with a better backup service, plus a fresh install and plugins with way less side-loads and a faster load in general post and page.

Even now working with a subdomain all previous link on Inkbunny, Twitter or Discord that linked to posts on the old site now are redirected properly to the new site, meaning that 4 years of links are still working even after the shutdown.

WHATS NEXT?

Right now we’re back into “Decaged!” with the last few pages and we’re preparing for the anniversary with a new homepage with character’s bios, a bit of lore of Allicot and some proper worldbuilding, plus the classic welcoming message from me. The members site will also have some QoL updates such as a very requested grid-style gallery page and more polished emails for new members, some pages had their documents updated and our payment service was updated with fresh data for the new site.