All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

  • @uis@lemm.ee
    link
    fedilink
    English
    116 months ago

    Meanwhile Kaspersky: *thinks if so incompetent people can even make antivirus at all*

  • @Mikina@programming.dev
    link
    fedilink
    English
    206 months ago

    I see a lot of hate ITT on kernel-level EDRs, which I wouldn’t say they deserve. Sure, for your own use, an AV is sufficient and you don’t need an EDR, but they make a world of difference. I work in cybersecurity doing Red Teamings, so my job is mostly about bypassing such solutions and making malware/actions within the network that avoids being detected by it as much as possible, and ever since EDRs started getting popular, my job got several leagues harder.

    The advantage of EDRs in comparison to AVs is that they can catch 0-days. AV will just look for signatures, a known pieces or snippets of malware code. EDR, on the other hand, looks for sequences of actions a process does, by scanning memory, logs and hooking syscalls. So, if for example you would make an entirely custom program that allocates memory as Read-Write-Execute, then load a crypto dll, unencrypt something into such memory, and then call a thread spawn syscall to spawn a thread on another process that runs it, and EDR would correlate such actions and get suspicious, while for regular AV, the code would probably look ok. Some EDRs even watch network packets and can catch suspicious communication, such as port scanning, large data extraction, or C2 communication.

    Sure, in an ideal world, you would have users that never run malware, and network that is impenetrable. But you still get at avarage few % of people running random binaries that came from phishing attempts, or around 50% people that fall for vishing attacks in your company. Having an EDR increases your chances to avoid such attack almost exponentionally, and I would say that the advantage it gives to EDRs that they are kernel-level is well worth it.

    I’m not defending CrowdStrike, they did mess up to the point where I bet that the amount of damages they caused worldwide is nowhere near the amount damages all cyberattacks they prevented would cause in total. But hating on kernel-level EDRs in general isn’t warranted here.

    Kernel-level anti-cheat, on the other hand, can go burn in hell, and I hope that something similar will eventually happen with one of them. Fuck kernel level anti-cheats.

  • @StaySquared@lemmy.world
    link
    fedilink
    English
    45
    edit-2
    6 months ago

    Been at work since 5AM… finally finished deleting the C-00000291*.sys file in CrowdStrike directory.

    182 machines total. Thankfully the process in of itself takes about 2-3 minutes. For virtual machines, it’s a bit of a pain, at least in this org.

    lmao I feel kinda bad for those companies that have 10k+ endpoints to do this to. Eff… that. Lot’s of immediate short term contract hires for that, I imagine.

  • Pudutr0n
    link
    fedilink
    English
    226 months ago

    This is a better article. It’s a CrowdStrike issue with an update (security software)

  • AutoTL;DRB
    link
    fedilink
    English
    56 months ago

    This is the best summary I could come up with:


    There are reports of IT outages affecting major institutions in Australia and internationally.

    The ABC is experiencing a major network outage, along with several other media outlets.

    Crowd-sourced website Downdetector is listing outages for Foxtel, National Australia Bank and Bendigo Bank.

    Follow our live blog as we bring you the latest updates.


    The original article contains 52 words, the summary contains 52 words. Saved 0%. I’m a bot and I’m open source!

  • @jedibob5@lemmy.world
    link
    fedilink
    English
    296 months ago

    Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half…

  • @solrize@lemmy.world
    link
    fedilink
    English
    96 months ago

    Xfinity H&I network it down so I can’t watch Star Trek. I get an error msg connection failure. Other channels work though.

  • Sʏʟᴇɴᴄᴇ
    link
    fedilink
    English
    666 months ago

    Yep, stuck at the airport currently. All flights grounded. All major grocery store chains and banks also impacted. Bad day to be a crowdstrike employee!

    • @iknowitwheniseeit@lemmynsfw.com
      link
      fedilink
      English
      56 months ago

      My flight was canceled. Luckily that was a partner airline. My actual airline rebooked me on a direct flight. Leaves 3 hours later and arrives earlier. Lower carbon footprint. So, except that I’m standing in queue so someone can inspect my documents it’s basically a win for me. 😆

  • Victor
    link
    fedilink
    English
    146 months ago

    If these affected systems are boot looping, how will they be fixed? Reinstall?

  • @richtellyard@lemmy.world
    link
    fedilink
    English
    956 months ago

    This is going to be a Big Deal for a whole lot of people. I don’t know all the companies and industries that use Crowdstrike but I might guess it will result in airline delays, banking outages, and hospital computer systems failing. Hopefully nobody gets hurt because of it.