• Nothing4You@programming.devEnglish
    23·
    19 days ago

    it’s not just lemmy.world.

    of the larger instances, the following have trouble sending activities to lemm.ee currently:

    i pinged @[email protected] on matrix about 30h ago already about the issues with federation from lemmynsfw.com, as it was the first one i noticed, but I haven’t heard back yet.

    • r00ty@kbin.life
      2·
      19 days ago

      Wait, how do they get that data remotely? I was looking at my instance vs world and I saw there’s like the +1 hour from a week or so ago when I upgraded to latest mbin lol.

      I guess they’re looking at common activities and when they appear on each?

      • Nothing4You@programming.devEnglish
        9·
        19 days ago

        lemmy has a public api that shows the federation queue state for all linked instances.

        it provides the internal numeric id of the last activity that was successfully sent to an instance, as well as the timestamp of the activity that was sent, and also when it was sent. it also includes data like how many times sending was unsuccessful since the last successful send. each instance only knows about its own outbound federation, but you can just collect this information from both sides to get the full picture.

        there is also https://phiresky.github.io/lemmy-federation-state/site to look at the details provided by a specific instance.

        • seang96@spgrn.comEnglish
          1·
          19 days ago

          This is a nice tool though using it I think triggered my IP to be flagged with Cloudflare when I was trying to fix an issue with my instance and lemmy.ml.

        • r00ty@kbin.life
          1·
          19 days ago

          That makes sense. So it’s showing me world’s federation with me and not the other way (since I’m not sure such info is available on mbin)

          • Nothing4You@programming.devEnglish
            31·
            19 days ago

            pretty much, yeah. lemmy has a persistent federation queue instead of fire and forget requests when activities get generated. this means activities can be retried if they fail. this allows for (theoretically) lossless federation even if an instance is down for maintenance or other reasons. if mbin has a similar system maybe they could expose that as well, but unless the system is fairly similar in the way it represents this data it will be challenging to integrate it in a view like this without having to create dedicated mbin dashboard.

            • r00ty@kbin.life
              2·
              19 days ago

              We can see it ourselves. We use rabbitmq for incoming (and maybe outgoing, it’s been a while since I looked at how it is) federation. So, you can see the queues there. For incoming (from rabbitmq) and outgoing there are also queues (symfony messenger) and these handle failures and can be configured and can be queried.

              After the upgrade I just took the default configuration again (because it seems queue names changed). But I used to have various rules setup in rabbitmq for retries and it took a fair few tries before the messages ended up in the proper “failed” queue (which needs manual action to retry). Some items you eventually need to clear (instances that just shutdown, or instances that lost their domain for example). They will never complete.

              But it’s not exposed in any way to my knowledge. Well unless people have their rabbitmq web interface open and without login of course.

  • jordanlund@lemmy.worldEnglish
    71·
    20 days ago

    I know there was a problem where federation was lagging by a couple of days, but I thought it was fixed? I’ll hit up the Admins.

  • Daemon Silverstein@thelemmy.clubEnglish
    5·
    19 days ago

    and the rest of the fediverse probably wont see this.

    As a The Lemmy Club user, I can properly see the post. Federation seems to be working okay.

  • r00ty@kbin.life
    3·
    19 days ago

    Looking at incoming request. .world is working OK for me. They seem to be batching stuff like I’ll get nothing for 30 seconds, then over 3 seconds like 50+ requests.

    Of course I don’t know if their queue is backed up and I’m getting delayed stuff. I’d need to stop processing and look into the incoming queue to see what they’re sending.

    Bit of an edit. Looking at incoming again I can see under newest items, an entry from world that was 11 minutes old. Oh I have an idea. I’ll see if this edit gets there in a timely manner.

    Spoiler alert, it was instant.

    Oh ignore me. It’s specifically between those two instances I guess.

      • r00ty@kbin.life
        3·
        19 days ago

        I think it must be hit/miss. Because I think those edits I made would have gone from my instance to world and then from world to .ee, and it was happening within seconds.

        So, presumably random stuff is being dropped or delayed?

        • 🇦🇺𝕄𝕦𝕟𝕥𝕖𝕕𝕔𝕣𝕠𝕔𝕕𝕚𝕝𝕖@lemm.eeOPEnglish
          22·
          19 days ago

          Yeah ur right. Everything going on here must go through .world as it is the community. Which makes it even weirder that we dont have .world users federaring out.

          My understanding of federation is ur events get sent to ur instance (kbin.life) ur instance sends those events to the instance of the relevent community (lemmy.world) the instance of the community sends that event to all instances subscribed to that community (lemm.ee, kbin.life, etc) my instance receives that event then notified me.

          So this conversion is goibg through .world just that .world users events seem not to be gettibg sent to .ee

          I think we need someone smarter than both of us to fix this

  • w3dd1e@lemm.eeEnglish
    3·
    19 days ago

    I see it? Maybe I just haven’t noticed but I think .world is fine for me?

    Edit: nvm you might be right. I just checked my posts in [email protected]. They normally get 60-100 up votes but the last one only got 3.

  • jordanlund@lemmy.worldEnglish
    31·
    19 days ago

    Admins have been talking about it, it looks like it has something to do with banning/unbanning users. Sometimes a ban/unban doesn’t want to propagate to .ee and that causes a logjam somehow.

    Smarter people than me are looking at it!