Alright, resolved with no downtime.
I didn't realize that the upgrade to Mastodon 2.6.0 *removed* the old rake tasks when tootctl was added. So, we weren't cleaning remote images. This caused the disk to fill up at 2018-12-19 15:10 UTC. I had mistakenly set up an alert for "Utilization" rather than "Disk Capacity", so Server Monitor never warned me about this.
* Add correct alerts (done)
* Update cron job to use tootctl (done)
* Ensure cron failures email me
@annika whoa, thanks, turns out I had the same issue on my instance, the disk was filling up due to the cronjob failing…
it just cleaned up about 20GB of old media
A place for the XOXO Festival community. Share your dreams, your struggles, your cat photos, or whatever else strikes your fancy, and see what everyone else is sharing.
This space is just for XOXO members. Never heard of Mastodon? Head over to joinmastodon.org to learn more and start posting.