Jan 24, 16:13 EST
It looks like we're out of the woods, and everything is back to operational.
We'll continue to keep an eye on things, and will follow up with a postmortem in a few days.
Jan 21, 16:06 EST
We are still seeing occasional outages. We continue to troubleshoot this issue with AWS.
Jan 21, 11:25 EST
A fix has been implemented and we are monitoring the results.
Jan 21, 07:34 EST
The path we tried to take to get things up and running didn't quite pan out the way AWS promised. We're still working with AWS to figure out the most expedient path forward, but as of right now, we don't have an exact ETA.
I'll update again once we know more.
Jan 21, 06:25 EST
A quick update on what's going on:
Some of our data volumes became sick tonight, leading to degraded performance, and eventually becoming unusable. We're currently working with AWS to see if it can be fixed.
We're also actively recovering the affected volumes from backup, to try and replace them, and see if that solves our issue.
If all goes well, this outage will last another 2-3 hours.
Jan 21, 02:27 EST
We are continuing to work on a fix for this issue.
Jan 21, 01:46 EST
The issue has been identified and a fix is being implemented.
Jan 21, 01:40 EST
We're experiencing a minor outage due to some release hiccups.
We're currently investigation the cause.
Jan 21, 00:17 EST