LiteLLM supply-chain incident

TeamPCP orchestrated one of the most sophisticated multi-ecosystem supply chain campaigns publicly documented to date. It cascaded through developer tooling and compromised LiteLLM and exposed how AI proxy services that concentrate API keys and cloud credentials become high-value collateral when supply chain attacks compromise upstream dependencies.

By: Peter Girnus, Fernando Tucci, Deep Patel, Simon Dulude, Ashish Verma, John Rainier Navato 2026/03/26 Read time: ( words)

On March 24, production systems running LiteLLM started dying, and engineers saw runaway processes, CPU pegged at 100%, containers killed by out-of-memory (OOM) errors. The stack traces pointed to the LiteLLM package, a popular Python package downloaded 3.4 million times per day that serves as a unified gateway to multiple LLM providers, was compromised on PyPI. Upon analysis, it was found that versions 1.82.7 and 1.82.8 contained malicious code that stole cloud credentials, SSH keys, and Kubernetes secrets.

The malicious versions deployed a three-stage payload: a credential harvester targeting over 50 categories of secrets, a Kubernetes lateral movement toolkit capable of compromising entire clusters, and a persistent backdoor providing ongoing remote code execution.

The compromise was not an isolated event. It was the latest link in a cascading supply chain campaign by a threat actor tracked as TeamPCP. This post traces the cascade from its origin, the open-source vulnerability scanner Trivy, and then presents our technical analysis of the LiteLLM payload.

TeamPCP orchestrated one of the most sophisticated multi-ecosystem supply chain campaigns publicly documented to date.

The campaign spanned PyPI, npm, Docker Hub, GitHub Actions, and OpenVSX in a single coordinated operation. While it did not specifically target AI infrastructure, the campaign's cascade through developer tooling caught LiteLLM in its blast radius and exposed how AI proxy services that concentrate API keys and cloud credentials become high-value collateral when supply chain attacks compromise upstream dependencies.

Key sections of this blog entry include a technical analysis of the malicious multi-stage payload and its impact on AI environments, a timeline and operational review of TeamPCP’s campaign, and a deep dive into how security tools themselves became attack vectors. TrendAI™ Research's analysis into the LiteLLM compromise also covers attribution challenges, gaps in public threat intelligence, and actionable defense strategies. Detailed indicators of compromise and MITRE ATT&CK mappings have been provided, but for an even more comprehensive understanding of this security incident, reach out to TrendAI™ Research for the full technical report. For our follow-up coverage on TeamPCP’s Telnyx Python SDK compromise, read our analysis here .

Trivy is an open-source vulnerability scanner developed by Aqua Security. It scans container images, filesystems, and infrastructure-as-code for security vulnerabilities, and it is integrated into the CI/CD pipelines of thousands of software projects via the trivy-action GitHub Action.

Security scanners are uniquely dangerous supply chain targets. By design, they require broad read access to the environments they scan, including environment variables, configuration files, and runner memory. When a scanner is compromised, it becomes a credential harvesting platform with legitimate access to secrets.

In late February 2026, an actor operating under the handle MegaGame10418 exploited a misconfigured pull_request_target workflow in Trivy's CI to exfiltrate the aqua-bot Personal Access Token (T1078 — Valid Accounts). Aqua Security disclosed the incident on March 1 and initiated credential rotation. However, according to Aqua's own post-incident analysis, the rotation "wasn't atomic and attackers may have been privy to refreshed tokens."

That gap proved decisive. On March 19 at 17:43 UTC, TeamPCP used still-valid credentials to force-push 76 of 77 release tags in the trivy-action repository, and all seven tags in setup-trivy, to malicious commits containing a multi-stage credential stealer (T1195.002 — Compromise Software Supply Chain). The malicious code scraped the Runner.Worker process memory for secrets, harvested cloud credentials and SSH keys from the filesystem, encrypted the bundle using AES-256-CBC with an RSA-4096 public key, and exfiltrated it to a typosquatted domain (scan[.]aquasecurtiy[.]org , resolving to 45[.]148[.]10[.]212 ). According to analysis by CrowdStrike, the legitimate Trivy scan still ran afterward, producing normal output — leaving no visible indication of compromise.

Figure 1 illustrates the chain that made the payload possible, which starts with a compromised security tool.

This is the meta-attack: a security scanner, the tool defenders rely on to catch supply chain compromises, became the entry point for a supply chain compromise. The Trivy compromise in GitHub Actions gave the attacker the keys to publish arbitrary versions of LiteLLM to PyPI. Everything that followed was exploitation of that initial foothold.

The lesson is uncomfortable but critical; your CI/CD security tooling has the same access as your deployment tooling. If it's compromised, everything downstream is exposed.

The attack didn't start with the .pth file. It started 13 minutes earlier with a cruder approach, and the rapid iteration tells a story the attacker's operational sophistication.

Version 1.82.7 (10:39 UTC): Payload injected directly into proxy_server.py, executes when the LiteLLM proxy starts. More targeted, but more visible in code review. Endor Labs identified three distinct payload iterations within proxy_server.py, suggesting the attacker was testing and refining in near-real-time.

Version 1.82.8 (10:52 UTC, 13 minutes later): Added the .pth file mechanism, executes at Python interpreter startup regardless of what's imported. Stealthier, broader impact. No import LiteLLM required. A simple command pip install LiteLLM==1.82.8 activated the payload on every subsequent Python process.

This 13-minute iteration, from a function-level injection to interpreter-level persistence, shows an attacker who understood Python's execution model deeply and adapted their delivery mechanism under operational pressure. Figure 2 illustrates how the iteration takes place.

LiteLLM is one of the most widely adopted open-source LLM proxy gateways in the AI/ML ecosystem. It sits between application code and model providers, OpenAI, Anthropic, Azure AI, and dozens of others, routing, load-balancing, and logging LLM API calls. Tens of thousands of developers and CI/CD pipelines depend on it daily. Version 1.82.8 shipped with something extra: a 34,628-byte file called LiteLLM_init.pth that silently exfiltrated every secret it could find to an attacker-controlled server.

A bug in the malicious payload, a fork bomb that spawned runaway processes, is what triggered the analysis. Without it, the credential stealer could have run silently for days or weeks before detection. The attacker's own coding error became the kill chain's weakest link.

A security researcher opened GitHub issue #24512 against BerriAI's LiteLLM repository with a simple, devastating subject line: "CRITICAL: Malicious LiteLLM_init.pth in LiteLLM 1.82.8 PyPI package, credential stealer." AI researcher Andrej Karpathy noted that the LiteLLM supply chain attack was quickly discovered due to a bug causing a system crash.

According to the security breakdown from FutureSearch , an attacker hijacked the maintainer accounts for the litellm project. They bypassed standard GitHub release protocols and pushed compromised versions directly to PyPI. Because litellm sits between developers and nearly every major LLM endpoint, it gets pulled in as a dependency by everything from basic scripts to advanced coding agents.

Within approximately three hours of the first community reports, PyPI yanked versions 1.82.7 and 1.82.8. The open-source community self-organized rapidly across GitHub, Twitter/X, and security vendor channels. BerriAI engaged Mandiant for incident response. The speed of the defensive response likely limited the real-world impact of the credential exfiltration. Though at the time of writing, no public confirmation exists whether stolen credentials were successfully used by the attacker before the takedown.

TrendAI™ Research has confirmed that both compromised versions, 1.82.7 and 1.82.8, use two attacker-controlled domains serving distinct purposes. Stolen credentials are encrypted with a hybrid RSA-4096 and AES-256 scheme and exfiltrated via HTTPS POST to models[.]litellm[.]cloud, a domain registered just one day before the attack.

A separate persistent backdoor polls checkmarx[.]zone every 50 minutes for second-stage payloads, abusing the trusted Checkmarx brand name to bypass DNS allowlists. Version 1.82.7 shares code lineage with an earlier supply chain campaign that targeted the same infrastructure, while 1.82.8 refines the payload with an identical dual-domain architecture.

The malicious versions were discovered not by security tooling, but by accident. Callum McMahon, a researcher at FutureSearch, was testing a Cursor MCP plugin that pulled in LiteLLM as a transitive dependency.

PyPI quarantined both versions approximately three hours after publication. Multiple downstream projects, including DSPy, MLflow, OpenHands, CrewAI, and Arize Phoenix, filed security PRs to pin away from the affected versions on the same day.

The malware is structured as a three-layer base64 encoded Python script. Each layer is decoded and executed at runtime, creating a chain of in-memory payloads that never touch disk as standalone files.

Table 1. The multi-layer malware architecture

The PyPI track was the highest-impact vector. Here's exactly what happened when a developer or CI/CD pipeline ran pip install LiteLLM==1.82.8 :

Figure 3 and the following sections provide more details on the execution flow.

Initial Launcher (Layer 0)

The entire file is a single Python line. When executed, it imports required modules and immediately spawns a child process to execute the base64-decoded Layer 1 payload.

Orchestrator (Layer 1)

Layer 1 defines the RSA public key, embeds , a base64-encoded payload and implements the run() entry point. A 4096-bit RSA public key is hardcoded for hybrid encryption of stolen data. The orchestrator creates a temporary directory and executes the collector script (Layer 2) as a subprocess, capturing all stolen data to a file. After the collector finishes and writes all stolen data to the temp file, the orchestrator encrypts it using a hybrid RSA+AES scheme. To do this, it first generates a random 32-byte AES session key using openssl rand. Then, it encrypts the collected data with AES-256-CBC + PBKDF2 key derivation using the session key. It then encrypts the session key with the embedded RSA-4096 public key using OAEP padding, and finally bundles the payload.enc + session.key.enc into tpcp.tar.gz. The encrypted bundle is then exfiltrated via HTTP POST to the attacker-controlled domain.

The credential harvester targets over 50 categories of sensitive data (T1552 — Unsecured Credentials):

System reconnaissance and SSH key and configuration theft

The collector begins by establishing helper functions for file reading and command execution, then immediately runs system identification commands. The emit() function reads any file and writes its contents to stdout with a marker header (=== /path ===). The run() function executes shell commands with a 10-second timeout and captures output.

All output is written as binary to stdout, which the orchestrator captures to the collected temp file. The script enumerates all user directories under / and /root, then reads every SSH private key, authorized key, and configuration file. It also targets SSH host keys from the system-level SSH configuration:

Cloud provider credential theft

The script reads AWS credential files from all users, dumps AWS environment variables, and queries EC2 Instance Metadata Service (IMDS) for IAM role credentials using both v1 and v2 token mechanisms.

When AWS credentials are found in environment variables, the script makes authenticated AWS SigV4 API calls to steal secrets from AWS Secrets Manager and SSM Parameter Store. The full SigV4 signing implementation is embedded in the script. It then queries IMDS v2 with a PUT request for a session token, followed by credential retrieval. With credentials in hand, it calls Secrets Manager ListSecrets + GetSecretValue and SSM DescribeParameters:

It steals all Google Cloud and Microsoft Azure credentials from the system including config files, cached tokens, service account keys, and related environment variables.

Kubernetes secret exfiltration

The script reads K8s config files and runs kubectl. It also reads the K8s service account token from the well-known mount path, then uses the K8s API to enumerate all secrets across all namespaces. API-based enumeration fetches secrets globally and per-namespace.