the instance is going down for a bit at 8 AM GMT on November 12th to finally put us behind iocaine, which should hopefully reduce or eliminate the amount of 503 errors we’re seeing (since all of them are scrapers pushing over the instance) and expose some metrics for admins trying to keep track of how our instance is performing

I’m also working on a bit of a treat for our regulars who’ve been putting up with the 503s for a while. that will probably be announced a bit after the downtime completes.

the good news: we’re now on the newest stable lemmy!

the bad news: federation feels a little off to me? sometimes this is a federation queue thing that resolves itself, sometimes it’s an indication of a problem.

things to test if you want to help out:

• see if you can see your posts on other lemmy and mastodon instances
• post here from other instances
• see if you can load communities, threads, and comments in non-local communities (this is a big one)
• see if you can load our communities from other instances and see up-to-date threads and comments
• make sure your own profile settings are as they should be
• if you aren’t getting email notifications and should be, let me know

I’ll push all my changes and post a full changelog once we know 0.19.12’s running stable!

our version of lemmy is old enough that clients like mlem are starting to break due to API drift, so I’m finally upgrading us to the latest stable version of lemmy. this will involve a bit of downtime and potentially a number of breakages; keep an eye out for anything that doesn’t look right after the upgrade and let us know!

after some extended downtime, I rolled out the following changes to our instance:

• pict-rs was migrated to version 0.4 then 0.5. this should hopefully fix an issue where pict-rs kept leaking TCP sockets and exhausting its resources, leading to our image uploads and downloads becoming non-functional. let me know if you run into any issues along those lines!
• NixOS was updated to 24.11.
• the instance’s storage was expanded by 100GB. this increased the monthly bill for our instance by €1.78 per month. to keep the bill low, I disabled an automated backup feature that became unnecessary when we started doing Restic backups.

I have one more thing I want to implement before our big Lemmy upgrade; I expect I should be able to fit it in tomorrow. I’ll update this thread with details when I start on it.

since we’ve been experiencing a few image cache breakages, I’m scheduling some maintenance for January 24th at 8AM GMT to upgrade our pict-rs version, increase the total amount of storage available to our production instance, and do a handful of other maintenance tasks. this won’t include a lemmy upgrade, but I plan to do one soon after this maintenance round. I anticipate the maintenance should take around 2-4 hours, but will post updates on the instance downtime page and Mastodon if anything changes.

we’ve exceeded the usage tier for our email sending API today (and they kindly didn’t email me to tell me that was the case until we were 300% over), so email notifications might be a bit spotty/non-working for a little bit. I’m working on figuring out what we should migrate to — I’m leaning towards AWS SES as by far the cheapest option, though I’m no Amazon fan and I’m open to other options as long as they’ve got an option to send with SMTP

big update, awful.systems is now a federated lemmy instance. let me know if anything looks broken! here’s what to expect:

• to pull up an awful.systems community on another instance, just paste that community’s URL into the other instance’s search bar
• federation with other lemmy instances should work, and probably kbin too? there’s no way I can find to pull in pre-federation posts on remote instances though, so send your friends here to read the backlogs
• we can’t federate with most of mastodon right now because lemmy doesn’t implement authorized_fetch, which is a best practice setting for mastodon instances. if your instance doesn’t use it, try entering something like @sneerclub@awful.systems into your mastodon search; lemmy communities are represented to mastodon as users
• this is pretty much an experimental thing so if we have to turn it off, I’ll send out another post
• reply to this post with ideas for moderation tools and instances you’d like to see blocked (and a reason why) and we’ll take action on anything that sounds like a good idea

federation was made possible by

• lemmy’s devs skipping their release process and not telling anyone 0.18.2 was released on friday? so we’re on 0.18.2 now
• updating all of the deployment cluster’s flake inputs just in case
• @dgerard@awful.systems shouting yolo