What happened.
On 19 July 2024 at 04:09 UTC, CrowdStrike released a content update to its Falcon sensor on Windows. It was reverted at 05:27 UTC, 78 minutes later. The systems in scope were Windows hosts running sensor version 7.11 or later that were online in that window and received the update. Microsoft estimated that 8.5 million Windows devices were affected, under one percent of all Windows machines.
The update asked the sensor for a 21st input. The sensor supplied 20. Reading the missing input crashed the system.
Six checks.
CrowdStrike's root-cause analysis of 6 August 2024 lists six findings. Each one gives you a check to run on your own production changes, whether they are code or configuration. Times are UTC. Where a fix was still planned on 6 August 2024, the text says so.
1. Check that a config and the code that fills it still match
The template type defined 21 input fields and the sensor code supplied 20. Nothing validated the field count when the sensor was compiled.
CrowdStrike put a compile-time check on the number of inputs into production on 27 July 2024. It also confirmed that no other template type supplied the wrong number, on any platform.
2. Test what your code does when a config asks for something it does not have
The code that reads the inputs had no runtime bounds check, so a read of the 21st entry in a 20-entry array went past the end.
CrowdStrike added a bounds check, and a check that the input count matches what the content expects, on 25 July 2024. It planned a hotfix to bring both, and the corrected count of 21, to Windows sensors 7.11 and later, generally available by 9 August 2024.
3. Use real values in every field of your tests
The automated tests used a wildcard in the 21st field of every test instance, so they did not expose the out-of-bounds read.
CrowdStrike now has automated tests with non-wildcard criteria for each field of every current template type, and requires them for all future ones.
4. Validate new content against what the running code provides
The content validator judged new content on the expectation of 21 inputs, but the code supplied 20. A new instance with a non-wildcard criterion for the 21st input passed validation and was sent to the interpreter.
CrowdStrike changed the validator to allow only a wildcard in the 21st field. It planned new checks so content cannot match on more fields than the code provides, to reach production by 19 August 2024.
5. Test every config change on its own, in the code that will run it
The template type was stress-tested with one test instance, and validating a later instance did not include testing it inside the content interpreter.
CrowdStrike's new test procedures test every new template instance, not only the first one made with a template type.
6. Roll config changes out in stages, with rollback
The report's sixth finding is that each template instance should be deployed in a staged rollout.
CrowdStrike added deployment layers and acceptance checks. New instances that pass canary testing are promoted ring by ring, with bake-in time between rings, or rolled back. Customers can choose where and when content updates deploy. CrowdStrike said it would keep adding finer control over those deployments, and release notes that customers can subscribe to.
Check that the teams who run the machines can choose when updates land.
How the crash happened.
The report says the crash came from three of these issues together: the validator expecting 21 inputs while the interpreter was given 20, the missing bounds check, and no test for a non-wildcard value in the 21st field. It also notes that the update was configuration data, not code or a kernel driver.
Sources.
- CrowdStrike, External Technical Root Cause Analysis: Channel File 291, 6 August 2024
- CrowdStrike, Preliminary Post Incident Review (PIR): Content Configuration Update Impacting the Falcon Sensor and the Windows Operating System (BSOD), 24 July 2024
- Microsoft, Helping our customers through the CrowdStrike outage, 20 July 2024
NOFire AI is the control layer that lets enterprises ship faster without breaking production. The Context & Control Model for Production.