The media partition is full. I'll take the server down momentarily to fix this.

Follow

Alright, resolved with no downtime.

I didn't realize that the upgrade to Mastodon 2.6.0 *removed* the old rake tasks when tootctl was added. So, we weren't cleaning remote images. This caused the disk to fill up at 2018-12-19 15:10 UTC. I had mistakenly set up an alert for "Utilization" rather than "Disk Capacity", so Server Monitor never warned me about this.

Remediation items:

* Add correct alerts (done)
* Update cron job to use tootctl (done)
* Ensure cron failures email me

My cron job for deleting old media wasn't running because the crontab $PATH didn't include the rbenv directory. Disk full alerts fired this morning when the disk hit 90% utilization. I've fixed the crontab and this cleaup should now run daily.

@annika whoa, thanks, turns out I had the same issue on my instance, the disk was filling up due to the cronjob failing…
it just cleaned up about 20GB of old media

Sign in to participate in the conversation
XOXO Zone

A place for the XOXO Festival community. Share your dreams, your struggles, your cat photos, or whatever else strikes your fancy, and see what everyone else is sharing.

This space is just for XOXO members. Never heard of Mastodon? Head over to joinmastodon.org to learn more and start posting.