CrowdStrike said a bug in its test software was the cause of a widespread IT crash that impacted millions of Windows systems globally.

In the most recent update on the incident, the cybersecurity company explained that on July 19, 2024, it issued a routine content configuration update for its Falcon platform's Windows sensor. This update aimed to gather telemetry on potential new threat techniques. However, the update inadvertently contained an error that caused Windows systems to crash.

The crash affected Windows hosts running sensor version 7.11 and above that were online and received the update between 04:09 UTC and 05:27 UTC on the same day. Systems not connected during this timeframe or those that came online after the defective update was reverted at 05:27 UTC were not impacted. Notably, Mac and Linux hosts were unaffected by the incident.

CrowdStrike’s Falcon Sensor utilizes "Sensor Content" to define its capabilities, with security content updates delivered in two ways: Sensor Content and Rapid Response Content. The issue arose from a Rapid Response Content update, which includes behavioral pattern-matching operations configured dynamically via Template Instances. These updates are meant to enhance threat detection and response without requiring changes to the sensor code.

The problematic update involved a new Template Instance for the InterProcessCommunication (IPC) Template Type. Although this Template Type underwent rigorous stress testing and was successfully deployed in previous instances, a bug in the Content Validator allowed the defective update to pass validation checks and be deployed into production, the company said.

Upon deployment, the defective content caused an out-of-bounds memory read in the Falcon sensor's Content Interpreter, leading to an unhandled exception that triggered a Blue Screen of Death (BSOD) on affected Windows systems.

The company said it took measures to prevent similar incidents in the future, including improving rapid response content testing, content validator enhancements, error handling improvements, and rapid response content deployment (staggered deployment strategy, enhanced monitoring, customer control).