ADVERTISEMENT

CrowdStrike crashing computers worldwide

KaH2qdv.jpeg
 
This is why I have our Autopatch updates staggered in Intune. Quality updates hit our IT department first prior to rolling out to the bulk of the organization, and then leadership is last (mostly because they whine like children if they are forced to reboot).

Unless there's a Zero Day exploit that needs to be patched. Then **** everyone's feelings, you're updating and rebooting.

Fortunately we aren't using any 3rd party auth services, so no Crowdstrike for us.

Central US Azure issues caused a couple minor issues for us, but all in all I had a chill Friday. I'm sure I won't be so lucky at some point but I'm going to enjoy this one.

This is an important point. It's not a great practice to set your mission critical systems to autoupdate en masse, especially for third party software that has drivers running in kernel space. Yes, it's CrowdStrike's fault, but you have to assume there will be occasional bugs and roll out updates cautiously.
 
  • Like
Reactions: like-woahh
Took me over 20 minutes on the phone with IT this morning to sort this out before anybody else woke up so i could avoid overnighting my laptop across the country to the office. All so I could run a 1 line statement in the command prompt
 
Took me over 20 minutes on the phone with IT this morning to sort this out before anybody else woke up so i could avoid overnighting my laptop across the country to the office. All so I could run a 1 line statement in the command prompt

Took me 72 minutes with my IT. I have a 7 letter account ID needed to find my account. I must have read it to her 7 times (using NATO phonetics) that she screwed up 7 times. That was honestly the most difficult part.

DEI. smdh.




















Kidding. Unless you think DEI also means offshoring IT support to Bangladesh/Mumbai/Manila
 
This is an important point. It's not a great practice to set your mission critical systems to autoupdate en masse, especially for third party software that has drivers running in kernel space. Yes, it's CrowdStrike's fault, but you have to assume there will be occasional bugs and roll out updates cautiously.

No company should ever run updates en masse to a bunch of prod servers. Hell we even patch sandbox environments with Windows before going live. I've seen enough .NET errors after patching to know better
 
No company should ever run updates en masse to a bunch of prod servers. Hell we even patch sandbox environments with Windows before going live. I've seen enough .NET errors after patching to know better
It’s definitely an indictment of the patch practices of the affected parties, but holy hell, how did such a backbreaking update get through Crowdstrike’s testing? It’s not look this took down an obscure iteration of Windows, it seems like it was universally destructive.
 
It’s definitely an indictment of the patch practices of the affected parties, but holy hell, how did such a backbreaking update get through Crowdstrike’s testing? It’s not look this took down an obscure iteration of Windows, it seems like it was universally destructive.

Not necessarily in this case. From what I understand this wasn't a full application update and more of a... virus definition update. These updates for Falcon bypass official policies and can access the kernel of Windows OS, which is why it was so devastating. Presumably these updates went via a Crowdstrike agent installed on the machines. We use Intune for example, so a patch or update for the application itself would need to be pushed via Intune, but they would be able to bypass Intune completely by pushing a definition update directly through the agent installed on the device. This really isn't unheard of, but to push code updates to the kernel level without them being verified is absolute madness.

Windows has a process of vetting any and all code running at the kernel level, but Crowdstrike also bypassed this, presumably to save time pushing these definition updates. So they might have had a driver with a valid cert, the code in the driver could be changed which wouldn't affect the validity of the cert on the outside.

This fella explains it very well:



Crowdstrike is already denying some of the above, but most people in the know think they're completely full of shit.
 
Last edited:
ADVERTISEMENT
ADVERTISEMENT