Banking On Local Expertise Weathers Global CrowdStrike Storm

On 19 July 2024, the global tech community faced one of its most unprecedented challenges, dubbed “the largest IT outage in history”. A botched update from global cybersecurity firm CrowdStrike triggered a cascading failure that affected approximately 8.5 million Microsoft Windows devices worldwide, including Capitec’s systems.

This outage tested the resilience and expertise of technology teams across the globe and saw South Africa’s leading digital bank experience disruptions across all its banking channels, from online banking to mobile app transactions and card payments. Initially, clients faced difficulties accessing various banking services, causing understandable frustration and concern.

Our clients’ needs in this situation was clear – unlimited access to their money anywhere, anytime. And – given our commitment to our client-first approach – we immediately jumped into action to solve for this need as quickly as possible. Our experience managing millions of clients, robust systems, and a dedicated team of more than 400 experts spanning multiple disciplines allowed us to restore our banking services within hours, becoming one of the first institutions globally to recover fully.

Here are the valuable lessons this experience has taught us, and the insights digitally enabled organisations can leverage to navigate the complex digital landscape while prioritising client needs.

The anatomy of a global tech crisis

To understand the magnitude of this event, we need to grasp its origins. CrowdStrike, a leading cybersecurity solutions provider, released an update to its Falcon Sensor vulnerability scanner. This seemingly routine update contained a critical flaw that caused Windows machines to crash, displaying the infamous “blue screen of death” and trapping millions of systems in an endless reboot cycle.

The impact was rapid and far-reaching. Airlines grounded flights, hospitals faced disruptions, and financial institutions worldwide grappled with system failures. Capitec and other banks and corporations worldwide experienced significant disruptions across their digital channels.

Building client trust: A speedy, transparent, and resilient response

Capitec is responsible for over 22 million clients – more than a third of the South African population. Their needs and lifestyles are the cornerstones of our offering. That’s why, over the last three years, we’ve invested R6.3 billion in re-platforming our systems, migrating our data to AWS Cloud services and building product.

The cloud was predominantly unaffected and was one of the most durable platforms during the outage. On-premise systems were affected the hardest. This significant investment proved crucial during the global CrowdStrike outage when the bank’s primary concern was restoring services as swiftly as possible to minimise client disruptions.

Additionally, thanks to an early warning monitoring system that checks Capitec’s services continuously, we could activate well-rehearsed crisis management protocols within minutes of detecting the CrowdStrike-related anomalies.

One of our first critical steps was to isolate our systems from further damage. We quickly blocked communication with CrowdStrike’s servers, preventing the problematic update from spreading to more of our infrastructure.

We executed this and subsequent decisive actions rapidly because of our strategically developed playbooks. Our efforts to regularly update these comprehensive crisis response plans allowed our team to accelerate the bank’s recovery through:

1. Rapid assessment: Capitec’s monitoring systems allowed it to identify the problem scope swiftly.
2. Cross-functional mobilisation: We assembled a team of over 300 Capitec IT specialists, spanning software engineering, infrastructure, cloud computing, cybersecurity, and network management, within minutes in online incident management rooms.
3. Prioritised recovery: The bank focused on restoring its most critical systems, particularly those directly impacting its clients, such as card transactions and ATM services.
4. Clear communication: Capitec prioritised transparent communication with its clients, keeping them informed through multiple channels about the nature of the problem and progress in resolving it.
5. Leveraging redundancy: The bank’s investment in multiple layers of security and redundant systems paid off, allowing it to maintain core services even as it worked to restore its systems.

Thanks to these efforts, Capitec could restore most of its critical banking services within two and a half hours of the initial outage. By early afternoon on 19 July, all the bank’s systems were back online, making us one of the first institutions globally to recover from this unprecedented event.

What we’ve learned regarding digital adoption

This incident offered several valuable lessons for operating in the current digital world:

1. Investing in robust monitoring: Our ability to detect and respond quickly to the outage was crucial. Implementing comprehensive, real-time monitoring of all critical systems is non-negotiable in today’s digital landscape.
2. Prioritising redundancy and resilience: Having multiple layers of security and redundant systems allowed us to maintain some services and recover others quickly. This approach should be a cornerstone of any robust IT strategy.
3. Developing and testing crisis playbooks: Our team’s ability to execute pre-planned crisis response strategies was instrumental in our rapid recovery. Regular drills and updates to organisations’ playbooks are essential.
4. Fostering a culture of agility: Hierarchical decision-making can slow response times in a crisis. Empowering our experts to make quick decisions based on their expertise was incredibly valuable.
5. Prioritising clear communication: Keeping stakeholders informed during the crisis was crucial. Having a clear communication strategy ready to deploy at a moment’s notice enabled us to respond quickly.
6. Regularly reviewing dependencies: This incident highlighted the risks of relying on single providers for critical services. Technology leaders should constantly assess our dependencies to mitigate risks in our technology supply chain.
7. Investing in diverse skill sets: Our ability to respond effectively was due to the diverse expertise within our IT team. Cultivating a wide range of technical skills within our organisation was essential.

Shaping a resilient digital ecosystem

The events of 19 July are a stark reminder of our digital interdependence and technology’s critical role in our daily lives. We view this incident as an opportunity to learn, grow, and strengthen our systems and processes.

We remain committed to leveraging technology to provide our clients with secure, reliable, and innovative banking solutions. We also believe that by sharing our experiences and insights, we can contribute to building a more resilient and robust tech ecosystem that can better serve clients in an increasingly digital world.

The tech and banking industries have shown remarkable resilience and adaptability in the face of adversity. By learning from this experience and implementing these lessons, we can emerge stronger and better prepared for future challenges.

• Wim De Bruyn, CIO at Capitec Bank