Staying Ahead of Emerging Threats

In late 2019, a new critical-severity vulnerability began to threaten widely used Citrix appliances. While the security community explored the issue and businesses scrambled to learn if they were exposed, our new Continuous Attack Surface Testing (CAST) service allowed us to identify affected assets, develop a working exploit, and offer remediation recommendations for our Cosmos clients – all before the Citrix patch or a public proof-of-concept exploit code were released.

WHAT'S Cosmos?

Cosmos, previously known as CAST (Continuous Attack Surface Testing), is Bishop Fox’s new managed service that empowers its seasoned operators (including me) to provide comprehensive penetration tests to clients on a continuous basis. With a real-time attacker’s view of an organization’s external perimeter and operationalized vulnerability data, CAST amplifies our ability to identify threats and advise our clients. We like to call it the Iron Man suit for hackers.

TIMELINE OF A THREAT

In mid-December 2019, NIST categorized a new critical-severity vulnerability (CVE-2019-19781) that allowed remote code execution on Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliances.

Figure 1: NIST categorization of this CVE with a 9.8 base score out of 10

Within three days of the vulnerability’s public advisory, Citrix released mitigation options for their users, but actual patches weren’t available until nearly a month after the CVE was publicly announced.

During this time, the CAST operators who were continuously testing client attack surfaces quickly determined that 50% of our CAST clients had vulnerable Citrix appliances deployed to their external perimeters. As the security community at large continued to investigate this vulnerability, our CAST team concluded that the initial publicly proposed exploit could cause excessive logging and result in a denial-of-service condition in the affected system. We considered this an unacceptable business risk.

CAST operator Caleb Gross had independently encountered this logging obstacle when testing the viability of an internally developed proof-of-concept payload, and chose to dig into the firmware for answers. Through his research, Caleb found a solution and modified the payload to avoid excessive logging so that testing would not disrupt client systems. Armed with a stable exploit, the CAST team could quickly discover the risks specific to each client and safely demonstrate each critical impact. CAST clients receiving our custom recommendations were motivated to apply mitigations immediately, knowing the true danger to their environment and the likelihood of a public exploit being released soon after.

On the morning of January 10, 2020, Bishop Fox notified its CAST clients about the specific threats that this CVE posed to their environments and our mitigation strategies to defend against it. At the end of that business day, other security teams released publicly available (but unstable) proof-of-concept exploits.

During his investigations, Caleb also discovered that Citrix appliances hosted on Amazon’s AWS infrastructure were configured to use the virtual machine’s instance ID as the password for the root account. If an attacker gained a foothold on one of these AWS instances, they could leverage a widely known internal metadata endpoint to obtain the instance ID, effectively providing instant root access on any device using the default configuration. CAST operators communicated this additional privilege escalation threat (not publicly known at the time) with our clients that day as well.

Figure 2: Operator Ori Zigindere reporting on the early warning that Cosmos customers received about this CVE

By moving quickly and digging deep, CAST operators found safer ways to test systems for this CVE and delivered concrete proof as to why clients should apply mitigations (and eventually, patches) to protect themselves against it. By the end of January, the threat that this CVE posed to Amazon Marketplace appliances had also been publicly shared by other security community members; Citrix then released patches that resolved the Marketplace attack vector as well as the directory traversal issue described in the original CVE.

THE SOLUTION: HUMAN EXPERTS VALIDATING RISKS IN REAL TIME

By combining real-time asset discovery, tracking, and continuous penetration testing, we were able to provide our clients with a deeper, contextual perspective of this potential breach. We notified and advised CAST customers about this critical Citrix vulnerability a full workday ahead of public exploits, giving them time to patch before attackers could take advantage.

ANOTHER DAY, ANOTHER VULN

The modern organization’s attack surface is volatile and ephemeral. Even for clients who were aware that they were using Citrix products, it was overwhelming to contend with a vulnerability that affected such a widespread and business-critical aspect of their organization. We worked as an extension of our client’s security team to keep them informed and ahead of the evolving threat as it emerged in real time.

For the CAST operators monitoring client assets every day, this Citrix CVE was just another day at the office: the rush of creating an inspired exploit, then improving it to ensure it was safe for the team to demonstrate to CAST clients. It’s yet another example of why continuous testing improves attack surface visibility, and why employing expert-led validation is critical to ensure the overall security of a client’s assets.

Special thanks to Caleb Gross (@noperator) and Barrett Darnell for their contributions to this article.