Hey lemmings!

I wanted to share a quick update about our recent performance issues and how I have addressed them.

The last 24h have been a bit rough for lemm.ee.

Last night, I spent some time debugging federation issues with lemmy.world. We managed to significantly improve the situation - lemmy.world content is now reaching lemm.ee with a very high success rate - but this has had the effect of increasing incoming federation traffic on our servers significantly.

Additionally, we have been seeing steadily increasing normal user traffic over the past week, which is awesome from a community standpoint, but of course means that our servers have to do more work to keep up with all the new people.

To top things off, today there appeared a badly configured instance in the network, which was effectively launching a DoS attack against lemm.ee for several hours. Most likely it was unintentional, but unfortunately the end result was a sudden increase in our server load.

All these factors combined resulted in a really bad experience for most lemm.ee users today. Page load times have consistently been spiking into as much as 10 seconds or more for the whole day:

In fact, a lot of page loads just timed out with errors.

Fortunately, it seems I have managed to clear up the problems!

I have put a bunch of mitigations in place, and after monitoring the situation for the past hour, it seems that our performance issues have been resolved for now. So hopefully, you can enjoy browsing lemm.ee again without it feeling like torture!

Here are specific steps I took:

  • I have doubled the hardware resources for our backend servers and database.
  • I purchased a Cloudflare pro subscription for lemm.ee for 1 year. This took out a considerable chunk of my budget for lemm.ee, but in return it will allow me to analyze and optimize our cache usage to a far greater extent. I am already seeing vastly reduced load times for cacheable content (try opening https://lemm.ee a few times in a row as a logged out user - it should be blazing fast now!)
  • I have configured a rate limiter which will prevent future DoS from the specific method that was used against us today.

Of course, all of the above is costly. Luckily, lemm.ee users have been very generous with donations in the month of June, and in fact a significant amount of donors have opted for monthly recurring contributions. This all gives me the confidence to increase our spending for now, and I am currently expecting to NOT increase my personal planned contribution of 150€/month, as the increased costs so far are entirely being covered by donations!

Let me take this opportunity to thank the sponsors who made the upgrades possible! All lemm.ee users are now enjoying better performance thanks to you, I could not have done it without you awesome people.

On a final note, I just want to say that I hope a lot of these issues can be solved by optimizations in Lemmy software itself in the future. I have been personally contributing several optimizations to the Lemmy codebase, and I know many others are focused on optimizations as well. Just throwing extra resources at the problem will probably not be a sustainable solution for very long 😅. But I am optimistic that we are moving in the right direction with the software changes, and we’ll be enjoying reduced resource needs before long.

That’s all I wanted to share today, I wish you all a great weekend!

  • rm_dash_r_star@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    1 year ago

    On a side note, really liking this 0.18.1 release candidate version, the 0.18.1 official release is going to be brilliant. The new compact view is beautiful and it scales with window width nicely. They just need to drop the post header size a bit and the compact view will be perfect. This release candidate seems to be pretty solid, only seeing fixes, no new bugs.

  • bool@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    You are awesome man. I wanted to wait until the instance matured before committing to a monthly donation but I am signing up now. You’re the best instance admin anyone can hope for. Glad to see your patch make it to 0.18

  • tryagain@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Alrighty, I had a browse around and I’ve settled on lemm.ee + Jerboa and it’s looking good. Thanks for all your hard work handling the influx of reddfugees like myself. This gives me hope ❤️

  • RoundSparrow @ .ee@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I have configured a rate limiter which will prevent future DoS from the specific method that was used against us today.

    For those of us running other Lemmy instances, what should we be looking for?

  • DeanFogg@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Been lurking off and on since the start of the evil reddit shenanigans. Finally set up on here using Connect. This is legit! Great job

  • Spzi@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Thanks for taking care, making upgrades, monitoring, fixing, contributing, and informing us so thoroughly.

    Also thanks to all the donors! This is all great to hear 😊

  • BillMurray@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I migrated a few hours ago when Apollo went down and have had a great experience so far! Nice work.

  • db0@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Can you tell me what you’ve cached? I’m not using cloudflare but I am using haproxy which has frontend caching builtin. It was next on my plan but if you share your caching setup I can try to replicate it on lemmy.dbzer0.com

    • sunaurus@lemm.eeOPM
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Mostly all images are served through a cache. I would like to also cache some static HTML (such as pages for unauthenticated users), but it breaks due to some users requesting these pages with an Accept header for an activitystream content-type, and I haven’t had time to figure out a solution for accounting for the content type in my cache key unfortunately 😅. But if you can do that easily in your cache then for sure you could also cache any static pages for a minute or so.

      • db0@lemmy.dbzer0.com
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Ye I can cache differently depending on headers. Surprised that caching images helps a lot since your pictrs is hosted in an independent box anyway