CrowdStrike Outage: Lessons from the July 2024 Incident

Background 

CrowdStrike is a global leader in cyber security. It is renowned for its innovative technologies. They are well known for their state-of-the-art cyber security products. The Falcon Sensor is a key product that is extensively utilised for endpoint security. It is used in a variety of sectors. This flagship solution utilises artificial intelligence, machine learning, and behavioural analytics. It provides protection against digital threats (CrowdStrike, 2020).

However, in July 2024, CrowdStrike’s systems went down globally. This impacted governments, corporations, and individual users. This occurred despite its reputation for robust security measures.  

This incident disrupted its operations and affected millions of devices worldwide. The event revealed serious flaws in CrowdStrike’s deployment plans and software update procedures. This case study examines the outage’s technical details, contributing factors, and lessons learned.    

Introduction 

The release of a major software update on July 19, 2024, marked the start of the incident. The flawed update was found in a file that is known to CrowdStrike as ‘Channel Files’ (Techtarget, 2024). These files provide the configuration updates for their service’s behavioral protections. The Channel File related to this incident was known as Channel File 291 (CrowdStrike, 2024a). It caused extensive service interruptions. System crashes started to occur on Falcon Sensor-powered devices, leaving many users without access to their systems for several hours.

Affected users reported issues in various parts of the world, indicating that this incident had an international impact.

In addition to its magnitude, the outage garnered attention for several underlying reasons, many of which were avoidable.   

The CrowdStrike Falcon provides endpoint devices with real-time threat detection and response services. This component, alongside the Falcon sensor, integrates and tracks data on endpoints to determine prospective security threats. The device functions within a larger protective infrastructure. It aims to safeguard clients against extensive cyber dangers. These include malware, ransomware, and sophisticated persistent threats (CrowdStrike, 2020). 

The outage was triggered by the flawed software update released to the Falcon Sensor, which introduced multiple technical defects. This caused millions of customers to experience service outages, due to data being processed incorrectly.  

Multiple industries across the globe reported being hit, showing the significant scale of the outage. As time went on, it became evident that the financial cost due to the downtime would be significant. Losses were estimated at $5.4 billion for Fortune 500 companies (CNN, 2024). Numerous prominent firms were forced to implement drastic measures to safeguard their devices while public sector entities faced operational delays. 

Equally, only a small percentage (10-20%) of the companies affected were thought to have been covered as part of their cyber security insurance policies (CNN, 2024).  

Timeline 

The incident started with the poor implementation of a software update on July 19, 2024. It began with the augmentation of the Falcon Sensor. The update aimed to enhance the Falcon Sensor. However, it inadvertently introduced a data field mismatch. It also made changes to the wildcard matching system. Wildcard matching is a method used to find items in a list or database. It uses special symbols like * or ? as placeholders for unknown parts. These issues caused widespread system failures (CrowdStrike, 2024a).

By mid-morning, users started to experience system outages accompanied by blue screens of death.

Figure 1: Blue Screen of Death (BSOD) (Wikipedia, 2022)

CrowdStrike’s tech team processed the data flaw and found the underlying problem the same day. In response to their findings, the company started the update rollback process. Although the partial recovery by evening hours was helpful, many users were still persistently troubled.

Although partial recovery achieved by the evening, many users continued to face disruptions. By the following day, July 20, CrowdStrike had successfully restored services for most users, though some regional issues persisted. The company released an official Root Cause Analysis report.

It outlined the causes of the outage and detailed steps for improving software deployment and testing processes (CrowdStrike, 2024a).   

Root Cause and Technical Analysis 

The root cause of the CrowdStrike outage was a flawed software update, which introduced several technical issues. The update caused a data field mismatch. The system expected 21 data fields but only received 20. This led to memory access errors and system crashes. Additionally, the update altered the wildcard matching system, which was responsible for interpreting data inputs (CrowdStrike, 2024a).

These changes were not adequately tested, particularly regarding how the wildcard matching system would perform under the different configuration.

As a result, the system failed to process data correctly, causing disruptions. Furthermore, the update was deployed globally without a staggered rollout, amplifying the issue across large numbers of systems at once (CrowdStrike, 2024a).

The incident was compounded by a lack of sufficient regression testing, which would have identified the errors, prior to deployment. These combined factors led to the widespread impact of the outage.   

The outage had significant effects across various areas. Businesses relying on CrowdStrike’s Falcon Sensor for computer security had to stop operations, causing major issues and delays. The system failure interfered with their ability to manage online threats, particularly in cyber security tasks.

Many individual users were unable to access their devices, leading to personal productivity losses. Financially, this incident caused notable losses, including costs for compensation, fines, and harm to their reputation.

CrowdStrike’s stock price fell more than 11%. Customers and investors questioned the company’s ability to maintain system reliability (CNBC, 2024). The company also faced challenges in maintaining customer trust, as some clients considered switching to different providers. 

Once the cause of the outage was identified, CrowdStrike acted swiftly to resolve the issue. They began to reverse the faulty update and worked on restoring the Falcon Sensor to its earlier, stable version. The process was gradual.

While systems began to recover by the evening of July 19, some users continued experiencing issues for several more hours. Besides these immediate recovery efforts, CrowdStrike issued a public statement. They acknowledged the issue, reassuring customers that the issue was being addressed (CrowdStrike, 2024b). By July 20, most services were restored. A detailed report was released. It explained the technical specifics of the incident.

Lessons Learned 

The company committed to improving its software update processes. They included implementing staged rollouts and enhancing testing procedures (CrowdStrike, 2024a). These measures aim to prevent similar problems in the future. This incident provides several important lessons for CrowdStrike and the broader cyber security industry.

The incident has provided several critical lessons for both CrowdStrike and the cyber security industry:

• Thorough Testing: Rigorous testing, including comprehensive regression testing, penetration testing, and real-world scenario validation, is essential prior to deployment. This ensures updates do not accidentally cause system failures.
• Staggered Rollouts: This risks associated with global update releases without staggered deployment were highlighted. Staggered updates could have limited the problem’s reach and reduced its overall impact.
• Incident Management Communications: Transparent and proactive communication during incident is crucial. While CrowdStrike eventually addressed the problem publicly, earlier and more detailed updates (where possible) could have alleviated customer concerns and frustrations.
• Enhance Monitoring: The need for improved monitoring systems was recongised, allowing for quicker identification of issues and mitigation before they escalate.

Key Takeaways 

The CrowdStrike outage of July 2024 serves as a crucial reminder of the importance of rigorous testing, careful software deployment, and effective communication in the cyber security sector.

The company was successful in restoring services. However, the incident exposed vulnerabilities in their software management processes.

These exposed weaknesses led to significant disruptions for customers.

By addressing these shortcomings, particularly in their testing and rollout procedures, CrowdStrike can better prevent similar failures in the future.

This incident highlights the need for a proactive approach to problem management. Transparency is crucial for maintaining customer trust. It is vital for preserving a company’s reputation in the competitive cyber security market. 

As professionals in the cyber security space, it’s crucial to reflect on these lessons and apply them in our practices to build resilient systems.

Join the conversation in the comments or connect with me directly – I’d love to hear your perspective!

References 

Techtarget (2024), Explaining the largest IT outage in history and what’s next. Available at: https://www.techtarget.com/whatis/feature/Explaining-the-largest-IT-outage-in-history-and-whats-next (Accessed: 23 March 2025). 

CrowdStrike (2024a), External Technical Root Cause Analysis — Channel File 291. Available at: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf (Accessed: 23 March 2025). 

CNBC (2024), Microsoft, CrowdStrike shares fall after major IT outage. Available at: https://edition.cnn.com/2024/07/24/tech/crowdstrike-outage-cost-cause/index.html#:~:text=All%20told%2C%20the%20outage%20may,lost%20productivity%20or%20reputational%20damage (Accessed: 23 March 2025). 

CNN (2024), CrowdStrike outage: What went wrong and how it was fixed. Available at: https://edition.cnn.com/2024/07/24/tech/crowdstrike-outage-cost-cause/index.html#:~:text=All%20told%2C%20the%20outage%20may,lost%20productivity%20or%20reputational%20damage (Accessed: 23 March 2025). 

CrowdStrike (2024b), To our customers and partners. Available at: https://www.crowdstrike.com/en-us/blog/to-our-customers-and-partners/ (Accessed: 23 March 2025). 

CrowdStrike (2020), Falcon Complete Infographic. Available at: https://www.crowdstrike.com/wp-content/uploads/2020/08/falcon-complete-infographic.pdf (Accessed: 23 March 2025). 

Wikipedia (2022), File:Bsodwindows10.png. Avaliable at: https://commons.wikimedia.org/wiki/File:Bsodwindows10.png#Summary (Accessed: 23 March 2025).