We’re aware of ongoing federation issues for activities being sent to us by lemmy.ml.

We’re currently working on the issue, but we don’t have an ETA right now.

Cloudflare is reporting 520 - Origin Error when lemmy.ml is trying to send us activities, but the requests don’t seem to properly arrive on our proxy server. This is working fine for federation with all other instances so far, but we have seen a few more requests not related to activity sending that seem to occasionally report the same error.

Right now we’re about 1.25 days behind lemmy.ml.

You can still manually resolve posts in lemmy.ml communities or comments by lemmy.ml users in our communities to make them show up here without waiting for federation, but this obviously is not something that will replace regular federation.

We’ll update this post when there is any new information available.

Update 2024-11-19 17:19 UTC:

Federation is resumed and we’re down to less than 5 hours lag, the remainder should be caught up soon.

The root cause is still not identified unfortunately.

Update 2024-11-23 00:24 UTC:

We’ve explored several different approaches to identify and/or mitigate the issue, which included replacing our primary load balancer with a new VM, updating HAproxy from the latest version packaged in Ubuntu 24.04 LTS to the latest upstream version, finding and removing a configuration option that may have prevented logging of certain errors, but we still haven’t really made any progress other than ruling out various potential issues.

We’re currently waiting for lemmy.ml admins to be available to reset federation failures at a time when we can start capturing some traffic to get more insights on the traffic that is hitting our load balancer, as the problem seems to be either between Cloudflare and our load balancer, or within the load balancer itself. Due to real life time constraints, we weren’t able to find a suitable time this evening, we expect to be able to continue with this tomorrow during the day.

As of this update we’re about 2.37 days behind lemmy.ml.

We are still not aware of similar issues on other instances.

Update 2024-11-25 12:29 UTC:

We have identified the underlying issue, where a backport for a bugfix resulting in crashes in certain circumstances was accidentally reverted when another backport was applied. We have applied this patch again and we’re receiving activities from lemmy.ml again. It may take an hour or so to catch up, but this time we should reliably be getting there again. We’re currently 4.77 days behind.

We still don’t have an explanation why the logs were missing in HAproxy after going through Cloudflare, but this shouldn’t cause any further federation issues.

Update 2024-11-25 14:31 UTC:

Federation has fully caught up again.

  • @Metz@lemmy.worldEnglish
    33•1 month ago

    Unfortunately not entirely true. They have the largest Linux Community. To be precise, the big one.

    ProgrammerHumor is there as well. The Open Source and Privacy communties are very bigs as well there. Completely dwaring their .world counterpart.

    I’m not a fan of .ml either but you can’t deny that they have this 3 parts (which very much represent the ideals behind lemmy) covered very well.

      • @WoahWoah@lemmy.world
        14•1 month ago

        They get smaller when less people federate with that clown-shoe of an instance. The idea that the community inertia in those three communities is forcing other instances to remain federated despite wide-spread dislike and disdain for .ml users is absurd. Burn the bridge. The more it gets defederated, the bigger similar communities on less toxic instances can and will grow.

        Could you imagine using that logic in any other context? “I hang out with a community of KKK members because three of their members know a lot about Linux and have a lot of Linux-involved friends.” Give me a break.

        • Draconic NEO
          3•1 month ago

          It happened that way with Beehaw. Beehaw used to be in the same position as lemmy.ml with some of their communities. But they decided to knee jerk defederate us and sh.itjust.works because we’re big. They stopped being the de-facto communities not just for us here but almost anywhere else. Defederation does curb this kind of network effect, and quickly too, especially when it causes the less active ones to inflate like crazy.

          • @LibertyLizard@slrpnk.net
            15•1 month ago

            Yeah. If anything the situation has become worse since that time. I was banned from the entire instance for a frank if somewhat critical discussion of Chinese history, even as I was trying to avoid contemporary politics because I knew that would get me in trouble.

            Anything other than fawning praise of the Chinese government is labeled racism and results in a ban.

        • @marcos@lemmy.world
          3•1 month ago

          Oh… I’ve tried to talk about fire retardants there once, just to have half of my words censored and my post becoming unreadable.

      • Lemminary
        10•1 month ago

        I don’t have a link because it’s been a while, but others and I have been piled-on by .ml and Hexbear users. In my case, 40+ people replied with nasty comments to a single comment I had made. And then I got straight-up banned from .ml because they misinterpreted what I was saying about communism. A mod cited whatever ambiguous and unrelated bullshit sidebar rule and I got the hammer anyway calling me a troll. But all the nasty circle-jerk comments stayed up. So fuck’em.

    • Draconic NEO
      2•1 month ago

      I think things like this can be helpful at curbing network effect. Which can only be really curbed by force. Beehaw used to have quite strong network effect in some communities too in the earlier days, but now, they have much less of a vice, and are actually much quieter than they were before.

      Might be a good idea to check out !de_ml@lemmy.blahaj.zone in these trying times. In case it’s a while before the issue is fixed.