On April 1, 2021, the Codecov team was alerted to a security event involving our Bash Uploader. The threat actor specifically targeted the Codecov Bash Uploader and used it to deliver a malicious payload to all Codecov users utilizing the Bash Uploader, The Codecov GitHub Action, The Codecov CircleCI Orb, and the Codecov Bitrise Step (collectively, the “Bash Uploaders”).
The team immediately worked to mitigate future impact of the incident by removing the malicious change from the Bash Uploader, and implementing controls to prevent it from being added again.
There were further impacts as the nature of the malicious code change extracted git remote origin URLs and environment variables from the environment where the maliciously altered Bash Uploader was executed. The nature of this attack and follow on impacts were detailed thoroughly in our Security Update on April 15, 2021.
The attacker leverage two key exploits:
Customers most likely experiencing this event were those that downloaded the Bash Uploader during the window when the threat actor had unauthorized access to the Bash Uploader and executed it.
Specifically affected customers received communication via individual emails and in-app notifications. Customers were instructed on how to assess their own situation by:
Additionally, from April 29th onward, Codecov included in-app notifications indicating specifically impacted organizations and repositories, and the names of potentially leaked environment variables.
The incident was first detected on April 1, 2021. A customer performing SHASUM checking on the Bash Uploader noticed a discrepancy between the SHA256 reported on GitHub and their own calculated SHA256 for the Bash Uploader. The customer raised this issue to us via our security email alias.
Upon discovering the situation, our team moved swiftly to understand the root cause and develop mitigation and remediation measures for customers.
Ongoing response to the incident was handled as follows:
We moved as quickly as possible in our response efforts, while (a) coordinating with federal law enforcement and cybersecurity agencies and (b) investigating the situation thoroughly so that we could provide accurate, actionable information to our customers.
To recover from the incident itself, the key taken by the attacker was revoked. Additionally, all production keys were audited and rotated. Furthermore, we updated all publicly accessible Codecov Docker images to use squashed and/or multistage builds. Docker images were also pushed over with new squash builds and all versions removed from Dockerhub.
With the help of our third-party forensic team, we also conducted a full investigation of our infrastructure, associated logs, and application logs to confirm that there were no other intrusions or inappropriate access to our systems.
Many remedial steps were taken as a result of the incident, and the following improvements were either made or are in the process of being made:
The Codecov team observed several points that we hope to with the industry:
The full story
This article is one source in a clustered incident — the cluster page carries the summary, timeline and every other outlet covering it.
