An unidentified malicious actor gained access to Coder’s Cloudflare infrastructure and added unauthorized IP addresses to the pool used for Coder’s module registry. These unauthorized IP addresses hosted a version of Coder’s registry that contained artifacts which included malicious code. The result was that for a short period of time Coder’s module registry website was serving a malicious version of our artifacts to a subset of our users. This malicious code was designed to identify a number of credentials and exfiltrate those to a remote server with a lookalike domain name compared to one Coder regularly uses.
Users who use the main Coder module registry (registry.coder.com) for their template source and who updated components during the time period when this code was available may have installed the malicious packages and been impacted by the credential stealing malicious software.
At this time we have no indication that any customer data maintained by Coder was impacted in any way.
At this time, we recommend running the steps below manually to verify if the terraform module versions and possibly purge any impacted module from the cache.
We are advising that any user who a create a new workspace template, update a template version, ran a dry run of template build or deployed a Coder workspace on a cluster that has terraform module caching disabled between 07:35 UTC on Monday, August 31st and 21:45 UTC on Monday, August 31st perform the following:
Determine if you may have been impacted
Due to the nature of this attack, where a malicious server that we do not own was communicating with impacted deployments, we do not have the ability to conclusively determine which users may have been affected. Therefore we recommend that you individually review and determine whether you may be impacted.
Review the timeline of any Coder workspace deployments that may have taken place as well as the Indicators of Compromise that are listed below to see if your deployments may have been impacted.
You may have been impacted if your Coder deployment downloaded a Coder Registry module between 07:35 UTC on Monday, August 31st and 21:45 UTC on Monday, August 31st. This primarily occurs when new templates or template versions are created, but can also occur independently during workspace creation when module caching is disabled on your template settings (this is enabled by default). As a result, workspaces created from templates that downloaded affected modules may also be impacted.
To identify potential cases of data loss, review firewall, proxy, DNS, and VPC flow logs for outbound traffic to coder-infra[.]com .
We have provided a SQL query you can run against your Coder deployment to identify potentially affected cached modules and template versions:
Manually clear any cached packages that may have been impacted
For deployed systems, be aware that these malicious templates may be cached in your deployment infrastructure. Ensure that you clear any cached packages from your systems before performing a fresh deployment.
Run the following query against your Coder deployment to remove potentially malicious modules from your Coder deployment’s cache:
Update your Coder deployment
At this time, the currently served version of Coder templates have been reviewed and validated to be free from malicious code. Once you have ensured that you have cleared any cached packages please update to the latest version of Coder (patches being released on Sept 1) to ensure that you have the most up to date and secure version.
We are preparing our version of Coder to include automatic remediation steps in addition to the normal updates for the product.
Rotate your Credentials
Proactively rotate any credentials that may have been accessible to your Coder deployment, in particular your workspace provisioners. This includes API keys related to your cloud infrastructure, AI tooling, CI/CD credentials, and any other similar credentials that may either be deployed as environment variables, included in your configuration files, or simply available in your terminal history.
Effects of Malicious Packages
There are two scenarios for when the affected registry modules would have run.
First is when a template is uploaded, updated or a dryrun of a template change is made. In this scenario the provisor does not have any user secrets or information passed into it. The risk in this scenario is leaking of environmental variables and secrets on the provisioner itself.
The second scenario is when a workspace build was run, this includes three additional secrets that are passed to the provisoner, the user’s OIDC token, a user ssh-key if ssh-key is configured for the user, and any external auth provider which is configured for the affected templates use, single time tokens, no refresh tokens are included.
Additionally, if you are running the provisioner as part of coderd and not as a separate service this likely also leaked coder configuration variables such as your database password, external auth providers and other configurations
To identify potential cases of data loss, review firewall, proxy, DNS, and VPC flow logs for outbound traffic to the untrusted domain coder-infra.com .
The full story
This article is one source in a clustered incident — the cluster page carries the summary, timeline and every other outlet covering it.
