Our Mastodon server stopped working after around 22:37 UTC time yesterday. It appears Mastodon spun out of control consuming lots of CPU, disk and memory. It locked up the server with a massive amount of I/O. There isn’t a lot of information about what happened since logging stopped persisting to storage shortly after it started. It was quickly detected by our service monitoring but we just weren’t around to fix it for 7 hours. It’s working again after a server restart but nothing was changed.
OS release announcement we posted several hours earlier never got processed and actually posted, so it appears something was wrong with it for at least around 4 hours before it stopped responding to requests. It seems to be working fine now.
We’re not sure what happened. It might have been a small scale denial of service attack of some kind which started disrupting the service and took a while to fully take it down…
We posted this OS release announcement again: