Image via cloudflare.com

Today when Software-as-a-Service (SaaS) has become the backbone of global enterprises, a stark reminder of its vulnerabilities came to light in November 2023. Cloudflare, a leader in internet security, was caught in the crosshairs of a sophisticated cyber attack. The cyber attack on Cloudflare, facilitated by compromised credentials from the Okta breach, serves as a reminder of the cascading effects in our interconnected digital supply chain.  It’s not just about securing your perimeter but ensuring that every link in your supply chain is equally fortified.

Who was breached?

In November 2023, Cloudflare disclosed a significant data breach, revealing that a nation-state actor infiltrated their self-hosted Atlassian server. The breach unfolded between November 14 and 17, involving reconnaissance activities and unauthorized access to Cloudflare’s internal wiki and bug database, both powered by Atlassian Confluence and Jira, respectively. The threat actor returned on November 22, establishing persistent access through ScriptRunner for Jira and advancing to Cloudflare’s source code management system, hosted on Atlassian Bitbucket. Although unsuccessful in accessing a console server linked to a São Paulo, Brazil, data center not yet in production, the attackers managed to view 36 Jira tickets and 202 wiki pages.

On November 16, the adversaries created an Atlassian account for sustained access, returning on November 20 to verify their continued presence. By November 22, the threat actor introduced the Silver Adversary Emulation Framework, ensuring persistent access to their self-hosted Atlassian server and enabling lateral movement. During the breach,120 code repositories were inspected, with 76 downloaded to their self-hosted Atlassian server, though no exfiltration occurred.

The attackers leveraged a Smartsheet service account to infiltrate Cloudflare’s Atlassian suite, prompting termination of the account within 35 minutes of detection on November 23. Additionally, the unauthorized user account created by the attackers was deactivated 48 minutes later, demonstrating Cloudflare’s swift response to the security incident.

How did this attack happen?

This breach involved the unauthorized use of stolen tokens and credentials, allowing the perpetrator to gain access to a limited amount of source code and documentation before being stopped. The actor acquired these credentials through a recent breach in Okta, resulting in the theft of thousands of credentials. Unfortunately, the organization failed to promptly rotate one service token and three service accounts following the breach. The compromised credentials included a Moveworks service token providing remote access to Atlassian systems, a Smartsheet account with administrative privileges in the Atlassian Jira instance, a Bitbucket service account with access to the Cloudflare source code management system, and an AWS environment devoid of access to the global network and lacking any customer or sensitive data.

How did Cloudflare respond?

Cloudflare has clarified that the security lapse was not attributed to Atlassian, AWS, Moveworks, or Smartsheet but resulted from a failure to rotate stolen credentials under the assumption of their non-use. Cloudflare, leveraging its zero-trust architecture, successfully contained and eliminated the infection. Through stringent access controls, firewall rules, and the implementation of hard security keys via their Zero Trust tools, the threat actor’s lateral movement was effectively limited, ensuring no services or global network configurations were compromised.

Recognizing the attacker’s intent to establish persistence, Cloudflare implemented a comprehensive remediation strategy. Despite confirming limited access, the company took extensive measures, including rotating over 5,000 production credentials, physically segmenting test and staging systems, conducting forensic triages on 4,893 systems, and reimagining and rebooting every machine in their global network, including those accessed by the threat actor and all Atlassian products (Jira, Confluence, and Bitbucket). Additional remediation efforts involved scrutinizing outdated software packages, identifying unused employee accounts, and securing potential secrets within Jira tickets or source code.

Cloudflare assured that the breach did not harm any customers or users. However, it sparked inquiries about the Okta breach, raising concerns across various services and companies affected. The company’s commitment to a comprehensive and swift response reflects its dedication to security and mitigating potential risks for future attacks.

Key Takeaways and Lessons

Customers are continually suffering from the consequences of data breaches of numerous organizations. With each breach public outrage seems to grow and expand. In the wake of these breaches individuals can start to learn lessons to improve their own security hygiene. Here are some things the average person can take away from this breach:

• Nation State Attackers Leveraging SaaS as an Entry Vector: This breach served as a stark reminder that nation-state actors increasingly exploit Software-as-a-Service (SaaS) platforms as a gateway into organizations. In this case, the threat actor infiltrated Cloudflare’s internal systems through breached Okta credentials, emphasizing the need for heightened vigilance and continuous third party risk monitoring. Highly specialized malware and hack tools are oftentimes no longer needed as SaaS creates an increasing attack surface for all organizations.

• The Crucial Role of Strong SaaS Incident Response: Cloudflare’s swift response was commendable. Upon notification of breached credentials from Okta they rotated thousands of credentials upon notification, but unfortunately missed a few service accounts and integrations. During the attack their security team promptly detected the threat within a week, much lower than the average breach identification time of 207 days1, then severed the attacker’s access within an hour and engaged CrowdStrike’s Forensic team for independent analysis. They were able to quickly respond and contain this incident in a timely manner due to their zero-trust architecture. Effective incident response surrounding SaaS platforms is non-negotiable in today’s threat landscape.

• Centralized User Management for Remediation: This breach highlighted the importance of centralized user management across SaaS platforms. If Cloudflare rotated stolen credentials promptly (following the Okta compromise in October 2023), the threat actor’s lateral movement could have been further restricted. While they did rotate most of the credentials,  they were still lacking the full visibility and tracking of the accounts left behind. Organizations who are not able to centrally track and manage user access management will struggle in the face of stolen credentials and other common threat vectors.

• Monitoring and Detection: Continuous monitoring and robust detection mechanisms are critical in SaaS environments. Cloudflare’s ability to detect anomalies and identify the threat actor’s activities allowed them to mitigate the impact swiftly. Organizations must invest in advanced monitoring tools to stay ahead of adversaries. Monitoring needs to go beyond just anomalous logins and encompass things like configuration changes, new integrations, and data exfiltration. Visibility into all actions surrounding SaaS platforms is key in the ever evolving threat landscape.

• The SaaS-to-SaaS API Connection Threat: The breach also highlighted the risks associated with SaaS-to-SaaS API connections and integrations. Organizations must rigorously assess these linkages to prevent unauthorized access. Many SaaS integrations evade security review and leave gaps in the security teams risk management and monitoring processes. These connections introduce the risk of losing sensitive data and should be closely reviewed and continuously monitored.

• Supply Chain and Third-Party Risk Monitoring: Cloudflare’s incident underscores the need for robust supply chain and third-party risk management. Vigilance extends beyond internal systems; organizations must scrutinize their partners and vendors. Third party risk management must be an end-to-end process moving beyond just pre-procurement third party risk reviews to configuration management, integration reviews, continuous monitoring, and proper offboarding. Many organizations are missing these key steps and are increasing their attack surface and creating major blind spots.

Conclusion

As we navigate the age of the SaaS enterprise, we need to treat SaaS like it is any other self hosted infrastructure or digital asset. These platforms store sensitive data and support key mission critical business processes and need to be secured like a critical asset. Organizations must truly understand the shared responsibility model and ensure they are implementing all the proper controls that are not supported by third parties. Building a SaaS governance framework assures that your data and processes are being protected. It builds end-to-end coverage from policy to response and recovery that creates defense in depth in a time where SaaS platforms are becoming targets for sophisticated threat actors. A strong SaaS security strategy is an often overlooked necessity that will continue to become more important as the digital age progresses.

To learn more about BluOcean’s SaaS Security and Privacy solutions visit: https://bluoceancyber.com/saas-security-and-privacy/ 

Resources

• Shweta Sharma. Nation-state actor used recent Okta compromises to hack into Cloudflare systems – CSO

• Ionut Arghire. Cloudflare Hacked by Suspected State-Sponsored Threat Actor – SecurityWeek

• Duncan Riley. Cloudflare Atlassian server hacked by suspected nation-state attacker – siliconANGLE

• Cloudflare Breach: Nation-State Hackers Access Source Code and Internal Docs – The Hacker News

• Jonathan Greig. Nation-state actor used stolen Okta credentials in Thanksgiving attack, Cloudflare says – The Record

• Onur Demirkol. Cloudflare Okta breach doesn’t have a big impact, company says – DATA CONOMY
  

‌Note: This blog post is based on information available as of February 5th, 2024, from various news sources. The situation might evolve, and readers are advised to stay updated through official channels.