Hi Beeple!

Here’s a vague version of events :

  • 11PM EST: Lemmy.world got hacked

  • 12:20AM EST: Blahaj.zone got hacked

  • 12:25AM EST: I shut down the server

  • 12:30AM EST: I make announcements to tell people about this

  • 12:45AM EST: I have an idea of what the problem is but there is no fix

  • 2:20AM EST: I go to sleep

  • 8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied

  • 11:40AM EST: We start testing things to make sure are working And well, now here we are.

If you have issues logging in or using an app:

  1. Log out if you somehow are still logged in

  2. Clear all cache, site data, etc.

  3. Hard refresh Beehaw using CTRL+F5

  4. Log back in.

If you still have issues, write to us at [email protected]

To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.

Oh yeah, in case, you haven’t, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw

  • frogman [he/him]@beehaw.orgEnglish
    51·
    1 year ago

    shutting down the server early was best. the nature of open source software is what allows these incidents to be mitigated as quickly as they are. thanks a lot to you guys, and to all of the team at Lemmy who worked to resolve this.

    heroes <3

  • alehel@beehaw.orgEnglish
    36·
    1 year ago

    Thank you for shutting down rather than “wait and see”! It was the right choice.

  • comicallycluttered@beehaw.orgEnglish
    29·
    1 year ago

    Agree with everyone else. Thanks for shutting it down.

    I’ll most likely do it anyway, but do you think password changes are necessary at this point?

    • Lionir [he/him]@beehaw.orgOPMEnglish
      31·
      1 year ago

      I don’t think this is necessary.

      We had no messages on our database that had the vulnerability though some were federated from blahaj in the aftermath. The JWT, which is your session token, was changed as well so it seems very unlikely to me that this needs to be changed. There’s no reason to believe the attack could’ve given access to passwords.

      • comicallycluttered@beehaw.orgEnglish
        12·
        1 year ago

        Thanks! Was just making sure.

        Anyway, really glad you guys took action as soon as you realized what was happening. Super appreciated.

      • jarfil@beehaw.orgEnglish
        5·
        1 year ago

        We had no messages on our database that had the vulnerability

        This is interesting. I actually commented about the use of emojis/emotes a couple days ago on a post on [email protected] made by a federated user from lemmy.one, that has since been removed (😕), but I still have the bookmarked comment in which I copied the raw embed for the remote emote image in the federated comment I was responding to.

        Do I understand it correctly, that the latest fixes to stop the code injection, will still allow remote image embedding, so something like an “emote picker extension to embed animated GIFs from a remote server and/or remote instance’s emoji list” would still be doable and wouldn’t pose any risk?

        Or would such picker still have to include measures to prevent offering embeds with potential exploits?

        • Lionir [he/him]@beehaw.orgOPMEnglish
          7·
          1 year ago

          Remote image embedding is not the issue, remote custom emojis would not have been an issue either. The issue, from my understanding, is that the way local emojis are rendered allowed for an XSS exploit.

          You can look at the PR which fixed this issue if you have a better understanding of these things than me : https://github.com/LemmyNet/lemmy-ui/pull/1897/

          I believe such a picker would be fine.

          • jarfil@beehaw.orgEnglish
            3·
            1 year ago

            I see, so the prior emoji handling rendered content directly from the comment, instead of making sure it was strictly what was defined for the local emoji; that was a weird choice. Now they’ve also added a sanitizer wrapper to all of it in: https://github.com/LemmyNet/lemmy-ui/pull/1906

            I guess the only downside of a picker that used the non-emoji image renderer, would be the loss of emoji CSS formatting.

            • tanglisha [she/her]@beehaw.orgEnglish
              3·
              1 year ago

              From what I can tell the whole point to the css class/formatting was controlling the size of the emojis. Depending on where they came from, I could see some being of random size and shape. Admins might not have the time or know-how to shrink them down, so css seems like a reasonable compromise as long as the files aren’t huge.

              I’m kind of bothered that the only fix seems to be on the frontend. Unfortunately, I haven’t been able to stick with Rust long enough to take a reasonable crack at figuring out how to help on the backend. Input and output sanitization should ideally be handled in both places.

              • jarfil@beehaw.orgEnglish
                1·
                1 year ago

                Lemmy’s backend is kind of curious, in that it does the bare minimum to move content around and add some metadata fields.

                For example, did you know that “deleting” a comment, only marks a “deleted: true” field, while the comment is still pushed in full to the frontend? Same thing happens with banned/mod removed comments, they just get marked as “removed: true” but otherwise still get pushed to the client, and the user can still edit them.

                All the display processing is done in the frontend, or whichever app you happen to use.

                • tanglisha [she/her]@beehaw.orgEnglish
                  0·
                  1 year ago

                  I can maybe see marking it as deleted in case someone wants to creat undelete functionality later. I don’t agree with it, but I can see why someone would do it.

                  It’s just weird to still push it to the frontend.

                  Same with the removed stuff. All of that should be handled on the backend and never even sent to the frontend. Sometimes the reason for deletion is something you don’t want getting grabbed by someone who is bored and poking around in developer tools, like doxxing information.

                  Since I don’t have the time to do anything about it, though, I guess I don’t have a place to complain. I have strong feelings about this stuff, but there’s a limit to the number of things a single person can work on. If I were to hop on an open source project this minute, it would be helping migrate Cursorless to an LSP.

      • abhibeckert@beehaw.orgEnglish
        5·
        1 year ago

        I don’t think this is necessary.

        I’d add that it’s basically useless. From what I’ve seen, resetting your password doesn’t even invalidate previously issued JWT tokens, which would be the only reason to do it. But of course, you’ve already reset them all and so has lemmy.world.

    • The Cuuuuube@beehaw.orgEnglish
      13·
      1 year ago

      To add onto what @Lionir said, you’ll never be wrong to change your password, even if much like in this case it isn’t warranted. For future reference, my recommendation is “if you have to ask, rotate your password.” Finding out later you didn’t have to is so much better than finding out later you should have

      • abhibeckert@beehaw.orgEnglish
        5·
        1 year ago

        I disagree - rotating passwords comes at a cost especially for people who don’t use a good password manager (and that is basically everyone). It’s security theatre and generally creates distrust between people offering security advice and the people who (hopefully) are listening.

        There are times when it should be done, but don’t do it without a reason.

  • fracture [he/him] @beehaw.orgEnglish
    15·
    1 year ago

    huge Ws, excellent work

    also, thanks for the Mastodon link, i wasn’t sure where to check on beehaw status during the outage

  • pwacata@beehaw.orgEnglish
    14·
    1 year ago

    Awesome response, and a great succinct postmortem. Thanks for doing what you do!

  • YourHeroes4Ghosts@beehaw.orgEnglish
    13·
    1 year ago

    Thank you for all you do, from what I was hearing I was in no way expecting you to have the site back up within 12 hours. Many kudos.

  • Lee@beehaw.orgEnglish
    12·
    1 year ago

    This is why I am on Beehaw. The Admins really care about the Instance and the content on it.

    That’s why I want to bring attention to the fact, that U can support them. https://opencollective.com/beehaw

    I am not a Admin, Mod or anything else. I just really like Beehaw and support them. And you should too.

  • gromnar@beehaw.orgEnglish
    12·
    1 year ago

    Good job on making the right call and preemptively shutting the server down. Thanks for being alert!

    • The Cuuuuube@beehaw.orgEnglish
      14·
      1 year ago

      Far more memorable than all the times the service was unavailable was all the times your data was breached. I’ll always prefer the service being down to having it up, and vulnerable

  • nlm@beehaw.orgEnglish
    11·
    1 year ago

    Great job keeping the site safe guys!

    Nice to see it back up again! It being offline was surprisingly palpable. Missed it!

    I’m guessing it’s probably not the last big thing that’s going to hit Lemmy instances in the future, everything still being in early development and all. Only things we can do is keep an eye out, have vigilant admins and plenty of backups!

    And patient users but we seem to have that. :)

  • emma@beehaw.orgEnglish
    11·
    1 year ago

    morning thought: I’ve definitely joined the right instance. (also the start from the assumption of good faith guidelines linked to in Gaywallet’s recent post)