How Code Execution Drives Key Risks in Agentic AI Systems

AI-driven applications are evolving from passive tools to agentic systems that generate code, make decisions, and take autonomous actions. This shift introduces a critical security challenge. When an AI system produces code, there must be strict controls on how and where that code is executed. Without these boundaries, an attacker can craft inputs that trick the AI into generating malicious code, which can run directly on the system.

Sanitization is often implemented as a primary defense mechanism. However, in agentic workflows, sanitization is insufficient. Attackers can craft prompts that evade filters, manipulate trusted library functions, and exploit model behaviors in ways that bypass traditional controls.

The NVIDIA AI red team approaches this as a systemic risk. LLM-generated code must be treated as untrusted output, and sandboxing is essential to contain its execution. This blog post presents a case study of a remote code execution (RCE) vulnerability identified in an AI-driven analytics pipeline, showing why sandboxing is a required security control in AI code execution workflows, not an optional enhancement.

Agentic AI systems are increasingly designed to translate user requests into code that is executed in real-time. The risk of this design is that AI-generated code is treated as trusted, but the LLM is following instructions from an untrusted input, and the resulting code must also be considered untrusted.

This involves a workflow where an LLM generates Python code that is executed directly by the application. Without proper isolation, this creates a pathway where crafted prompts can escalate into RCE.

While this example was identified during an evaluation of an analytics workflow, the core issue isn’t specific to a single integration or library. It’s a systemic pattern that affects any system executing AI-generated code without execution boundaries.

Sanitization techniques, such as filtering or modifying code before execution, are often implemented to mitigate this risk. However, sanitization is inherently limited. Attackers can craft inputs that exploit trusted library functions, evade static filters, and manipulate runtime behaviors in ways that sanitization cannot predict.

This repeating pattern follows a familiar chain:

Containment is the only scalable solution. Sandboxing the execution environment prevents AI-generated code from impacting system-wide resources, limiting the blast radius even if sanitization fails.

During a routine security evaluation, the NVIDIA AI Red Team reviewed an internal analytics workflow that used a third-party library to transform natural language queries into Python code for execution. The design pattern follows typical agentic AI workflows. A user’s input is sent to an LLM. The LLM then creates code to meet the request. This code is executed by the application.

Initial reviews confirmed that the library implemented code sanitization measures intended to restrict dangerous operations. However, deeper analysis revealed that these controls could be bypassed by calling functions from untrusted libraries that were imported by the trusted libraries. The issue stemmed from the challenge of controlling dynamically generated code with static sanitization techniques.

The identified vulnerability (tracked as CVE-2024-12366 ) illustrated a broader problem:

The AI red team collaborated with the library maintainers to disclose the findings responsibly and align on mitigation strategies. The engagement emphasized a shift from patching specific bypass techniques to implementing structural safeguards like sandboxing.

Sanitization is often the first response when securing systems that execute AI-generated code. However, as shown in the case study, sanitization alone is insufficient. Attackers can continuously craft inputs that evade filters, exploit runtime behaviors, or chain trusted functions to achieve execution.

The only reliable boundary is sandboxing the code execution environment. By isolating each execution instance, sandboxing ensures that any malicious or unintended code path is contained, limiting impact to a single session or user context.

Following the disclosure, the library maintainers introduced additional mitigations, including an Advanced Security Agent that attempts to verify code safety using LLM-based checks. While these enhancements add layers of defense, they remain susceptible to bypasses due to the inherent complexity of constraining AI-generated code.

The maintainers also provided a sandbox extension, enabling developers to execute AI-generated code within containerized environments. This structural control reduces risk by decoupling code execution from the application’s core environment.

The broader lesson is clear:

For organizations deploying AI-driven workflows that involve dynamic code execution, sandboxing must be a default design principle. While operational trade-offs exist, the security benefits of containing untrusted code far outweigh the risks of an unbounded execution path.

The security risks highlighted in this case study aren’t limited to a single library or integration. As AI systems take on more autonomous decision-making and code generation tasks, similar vulnerabilities will surface across the ecosystem.

Several key lessons emerge for teams building AI-driven applications:

As AI becomes deeply embedded in enterprise workflows, the industry must evolve its security practices. Building containment-first architectures ensures that AI-driven innovation can scale safely.

The NVIDIA AI red team thanks the PandasAI maintainers for their responsiveness and collaboration throughout the disclosure process. Their engagement in developing and releasing mitigation strategies reflects a shared commitment to strengthening security across the AI ecosystem.

We also acknowledge CERT/CC for supporting the coordination and CVE issuance process.