crazypeople.online admin. Feel free to reach out any time with questions, comments, suggestions, or concerns. DM for fastest response, or tag me in relevant posts.
Please use the report function for undesirable content so that community moderators are also notified and can address it quickly.
- 30 Posts
- 25 Comments
And we’re back up on 0.19.11. That’s enough instability for one day. We’ll come back to this another day once we have the network issue sorted.
Sorry again for the downtime for those of you using the webui, and for the repeated restarts for our app users. As always thank you for your patience and understanding.
Unfortunately it looks like we’re going to need to roll back to 0.19.11 for now. We’re seeing some network errors when trying to access the site from the webui on the 0.19.12 build that are going to take more time to investigate. Apologies, these things happen from time to time.
It took a little longer than expected to come back online, but we’re back now and things look normal. Things might be a touch slow for the next couple hours as things sync up again. Long term performance should be much more stable with fewer dips in performance when new communities are added or a new instance discovers ours.
Thank you again for your patience. We’re doing our best to stay ahead of the performance curve so our members can have the smoothest experience we can offer.
You’re welcome! There’s a lot of learning as we go and some things will always be out of our control.
Keeping everything running smoothly has been more hands on than expected with a managed hosting service. We have a good working relationship with our host but they are a relatively small operation so fixes and upgrades aren’t as seamless as they probably could be. Communicating downtime windows could also use some work as we’ve recently seen. Still, they do a great job responding to our concerns and helping with things that require more heavy lifting than we’re comfortable with.
I hope people are finding it an enjoyable place to spend their time online despite the hiccups here and there.
The upgrades are complete and so far everything looks good. We’re seeing vastly increased disk throughput and less time spent hanging while waiting for data. The instance should be feel more responsive more often now. I’m going to leave the pinned post up for now just in case anything else comes up, but I’m cautiously optimistic we are in the clear now.
This has been rescheduled yet again. Sorry for the on again off again updates.
Finally more information to pass along-
Our host has scheduled the maintenance window for 11pm-7am UTC tomorrow. It is likely the site will be completely unreachable for some portion of this time, and intermittently available for the majority of the maintenance window. Our best guess is that from 5am-6am UTC the site will be completely offline as the hardware upgrades and data migration take place.
The upgrade is causing some problems with our test instance and is on hold for now while we get them sorted out. Hopefully we can get it sorted and push the upgrade through here shortly.
A further update after more discussion with our host-
The major factor at play here is the increase in scrapers/bots/AI traffic to servers hosting federated content. There are large waves of bot requests that are pushing the postgres service to capacity across multiple customer sites within our host’s infrastructure. Our host blocks most scrapers automatically, and is actively working to block new bots to minimize the impact of the increased traffic as much as possible. Despite those efforts we’re still experiencing some problems here and there that we hope to have solved after the upgrades this weekend.
Again, thank you for understanding.
There have been a few short (less than 5 minute) service interruptions over the last couple days related to the coming upgrade at our host. We often aren’t given any warning if the downtime is expected to be very brief. Apologies if the experience is less than stellar. That isn’t what we want to provide.
Hopefully things will be running smoother than ever after this weekends maintenance.
The adjustments have been completed and everything looks good. The next expected downtime will be for our storage upgrades on June 14th.
Adjustments are done and everything looks to be back up as expected.
UPDATE: This has been pushed back one week by our host as there are a wider range of upgrades, mostly unrelated to our instance, being done at the same time to minimize downtime.
The new config wasn’t working as expected so we rolled back to our old email setup for now. Sorry for the minor disruption. Registrations are enabled again. We’ll come back to this project soon once we’ve got some of the underlying issues sorted.
The upgrades are complete and everything is back up and running. Please report anything out of the ordinary in terms of site performance.
Thank you.
Unpinning this since it’s been a couple days with no reported issues. If you run into anything just let us know.
Managing Lemmy Blocks
Lemmy offers multiple ways to block users and content from your feed. They can be accessed in the settings as seen below. App users will need to find these settings within their app of choice or use the web UI to make the changes.
How To Block Content
Open the settings menu in the top right.
Navigate to the Blocks tab.
Search for the user, community, or instance you want to block. Click the search result to confirm the selection.
Verify your selection was added to the block list.
How To Remove A Block
Open the settings menu in the top right.
Navigate to the Blocks tab.
Find the block you want to remove on the block list.
Click the “x”. A confirmation will appear in the bottom left.
How To Block Or Blur NSFW Content
Open the user settings menu from the top right.
Scroll down to the NSFW display options and select accordingly.
Click the Save button at the bottom of the page to finalize the settings.
And we’re back. Please report any issues you encounter.
We’re beginning the update now. The instance will go down in just a moment and be back in a few minutes. See you soon.
I haven’t had time to dig into it deeper yet and didn’t want to leave things in a mess for long yesterday. Everything appeared to update successfully but when the UI was updated and restarted it had several problems that made it unusable. Anything that uses the API had no trouble reaching the site after the upgrade which leads me to believe there’s an issue specific to the webui and how our instance is currently configured/hosted. I don’t want to speculate too much without more information. The setup at our host isn’t typical for most Lemmy instances and occasionally we run into issues that don’t impact instances run from a single server or from another host.