Read the full paper

The weaponization of Large Language Models (LLMs) for automated malware generation poses an existential threat to conventional detection paradigms. AI-generated malware exhibits polymorphic, metamorphic, and context-aware evasion capabilities that render signature-based and shallow heuristic defenses obsolete. This paper introduces CogniCrypt , a novel hybrid analysis framework that synergistically combines concolic execution with LLM-augmented path prioritization and deep-learning-based vulnerability classification to detect zero-day AI-generated malware with provable guarantees. We formalize the detection problem within a first-order temporal logic over program execution traces, define a lattice-theoretic abstraction for path constraint spaces, and prove both the soundness and relative completeness of our detection algorithm, assuming classifier correctness. The framework introduces three novel algorithms: (i) an LLM-guided concolic exploration strategy that reduces the average number of explored paths by 73.2% compared to depth-first while maintaining equivalent malicious-path coverage; (ii) a transformer-based path-constraint classifier trained on symbolic execution traces; and (iii) a feedback loop that iteratively refines the LLM’s prioritization policy using reinforcement learning from detection outcomes. We provide a comprehensive implementation built upon angr 9.2, Z3 4.12, Hugging Face Transformers 4.38, and PyTorch 2.2, with full configuration details enabling reproducibility. Experimental evaluation on the EMBER, Malimg, SOREL-20M, and a novel AI-Gen-Malware benchmark comprising 2,500 LLM-synthesized samples demonstrates that CogniCrypt achieves 98.7% accuracy on conventional malware and 97.5% accuracy on AI-generated threats, outperforming ClamAV, YARA, MalConv, and EMBER-GBDT baselines by margins of 8.4–52.2 percentage points on AI-generated samples.

The cybersecurity landscape is undergoing a fundamental transformation driven by the dual-use nature of Large Language Models (LLMs). While LLMs have accelerated legitimate software development through code generation, refactoring, and automated testing [ 1 ] , adversaries have simultaneously exploited these capabilities to produce sophisticated malware at unprecedented scale and velocity [ 2 , 3 ] . Recent threat intelligence reports document a 135% year-over-year increase in AI-assisted cyberattacks, with LLM-generated payloads exhibiting polymorphic behavior, semantic-level obfuscation, and adaptive evasion strategies that defeat traditional signature-based and static-heuristic defenses [ 4 ] .

The fundamental challenge posed by AI-generated malware is threefold. First, LLMs can produce functionally equivalent but syntactically diverse variants of the same exploit, defeating hash-based and pattern-matching detectors. Second, AI-generated code can embed trigger conditions that activate malicious behavior only under specific environmental contexts, evading sandbox-based dynamic analysis. Third, LLMs can iteratively refine evasion strategies by analyzing detection feedback, creating an adversarial arms race that static defense postures cannot sustain.

Concolic execution,a portmanteau of con crete and symb olic execution,offers a principled approach to this challenge by systematically exploring program execution paths through the interplay of concrete test inputs and symbolic constraint solving [ 5 , 6 ] . By maintaining both a concrete execution state and a symbolic path constraint, concolic engines can reason the conditions under which specific program behaviors manifest, including latent malicious behaviors hidden behind opaque predicates and environmental checks. However, the well-known path explosion problem ,where the number of feasible paths grows exponentially with program size and branching complexity,has historically limited the scalability of concolic analysis for real-world malware detection [ 7 , 8 ] .

This paper introduces CogniCrypt , a framework that resolves the scalability limitation by employing an LLM as an intelligent path oracle that guides the concolic engine toward execution paths with high malicious potential. The key insight is that LLMs, having been pre-trained on vast corpora of source code and security advisories, possess an implicit model of “suspicious” program behavior that can be leveraged to prioritize the exploration of paths most likely to reveal malicious intent. CogniCrypt further incorporates a transformer-based path constraint classifier that maps symbolic execution traces to maliciousness scores, and a reinforcement learning feedback loop that continuously improves the LLM’s prioritization policy based on detection outcomes.

Contributions. This paper makes the following contributions:

Formal Framework: We define a first-order temporal logic ℒ CogniCrypt \mathcal{L}_{\text{CogniCrypt}} over program execution traces and establish a lattice-theoretic abstraction of the path constraint space. We prove the soundness (no false negatives under the threat model) and relative completeness (detection of all malicious paths reachable within a bounded exploration budget) of the CogniCrypt detection algorithm (Section 3 ).

Novel Algorithms: We present three tightly integrated algorithms: LLM-Guided Concolic Exploration (Algorithm 1 ), Transformer-Based Path Constraint Classification (Algorithm 2 ), and Reinforcement-Learning-Based Policy Refinement (Algorithm 3 ) (Section 4 ).

Comprehensive Implementation: We provide a fully reproducible implementation built on angr , Z3 , PyTorch , and Hugging Face Transformers , with detailed configuration, hyperparameter settings, and deployment instructions (Section 5 ).

Extensive Evaluation: We evaluate CogniCrypt on four benchmarks,EMBER [ 17 ] , Malimg [ 18 ] , SOREL-20M [ 19 ] , and a novel AI-Gen-Malware dataset,demonstrating state-of-the-art performance, particularly on AI-generated threats (Section 6 ).

Symbolic execution was introduced by King [ 9 ] and has since become a cornerstone of program analysis. DART [ 6 ] and CUTE [ 5 ] pioneered concolic (dynamic symbolic) execution, combining concrete execution with symbolic constraint solving to achieve higher path coverage than pure symbolic approaches. KLEE [ 7 ] demonstrated the scalability of symbolic execution on real-world systems software by leveraging the LLVM intermediate representation. S2E [ 10 ] introduced selective symbolic execution, enabling analysts to focus on specific code regions within full-system emulation. More recently, angr [ 11 ] provided a comprehensive Python-based binary analysis platform supporting both symbolic and concolic execution, while Triton [ 12 ] offered a lightweight dynamic binary analysis framework with taint tracking and symbolic execution capabilities.

In the malware analysis domain, Moser et al. [ 13 ] applied symbolic execution to explore multiple execution paths in malware samples, revealing hidden behaviors triggered by environmental conditions. Brumley et al. [ 14 ] used symbolic execution for automatic patch-based exploit generation. Vouvoutsis et al. [ 15 ] recently demonstrated that symbolic execution can complement sandbox analysis to detect new malware strains. However, none of these works address the specific challenge of AI-generated malware or incorporate LLM-based guidance.

Machine learning approaches to malware detection have evolved from shallow models operating on hand-crafted features to deep learning architectures processing raw binary data. Raff et al. [ 16 ] introduced MalConv, a convolutional neural network that classifies PE files directly from raw bytes. Anderson and Roth [ 17 ] released the EMBER dataset and demonstrated the effectiveness of gradient-boosted decision trees (GBDT) on engineered PE features. Nataraj et al. [ 18 ] proposed visualizing malware binaries as grayscale images and applying computer vision techniques for classification.

More recently, transformer-based architectures have been applied to malware detection. Li et al. [ 20 ] proposed MalBERT, which fine-tunes BERT on disassembled malware code for family classification. Hossain et al. [ 21 ] demonstrated the use of Mixtral LLM for detecting malicious Java code. Al-Karaki et al. [ 22 ] provided a comprehensive framework for LLM-based malware detection, identifying key challenges including prompt engineering, context window limitations, and adversarial robustness.

The emergence of AI-generated malware represents a paradigm shift in the threat landscape. Pa et al. [ 2 ] demonstrated that ChatGPT can generate functional malware when prompted with carefully crafted instructions. Gupta and Sharma [ 3 ] showed that LLMs can produce polymorphic malware variants that evade signature-based detection. Beckerich et al. [ 23 ] introduced RatGPT, demonstrating automated phishing and C2 infrastructure generation. These works underscore the urgent need for detection techniques specifically designed to counter AI-generated threats.

Several works have explored the combination of symbolic execution with machine learning. Learch [ 24 ] used reinforcement learning to guide symbolic execution path selection in KLEE. However, no prior work has combined concolic execution with LLM-based guidance specifically for the detection of AI-generated malware, which is the unique contribution of CogniCrypt.

A program P P is modeled as a labeled transition system 𝒯 P = ( Σ , Σ 0 , ℐ , → , 𝒪 ) \mathcal{T}_{P}=(\Sigma,\Sigma_{0},\mathcal{I},\rightarrow,\mathcal{O}) where:

Σ \Sigma is a finite set of program states, where each state σ = ( ℓ , μ , ρ ) ∈ Σ \sigma=(\ell,\mu,\rho)\in\Sigma consists of a program location ℓ ∈ ℒ \ell\in\mathcal{L} , a memory map μ : 𝒜 → 𝒱 \mu:\mathcal{A}\rightarrow\mathcal{V} , and a register file ρ : ℛ → 𝒱 \rho:\mathcal{R}\rightarrow\mathcal{V} ;

Σ 0 ⊆ Σ \Sigma_{0}\subseteq\Sigma is the set of initial states;

ℐ \mathcal{I} is the input domain;

→ ⊆ Σ × Σ \rightarrow\subseteq\Sigma\times\Sigma is the transition relation;

𝒪 ⊆ Σ \mathcal{O}\subseteq\Sigma is the set of observable (output) states.

An execution trace τ = σ 0 σ 1 ⋯ σ n \tau=\sigma_{0}\sigma_{1}\cdots\sigma_{n} is a finite sequence of states such that σ 0 ∈ Σ 0 \sigma_{0}\in\Sigma_{0} and σ i → σ i + 1 \sigma_{i}\rightarrow\sigma_{i+1} for all 0 ≤ i ω ⁡ ( π b ) \omega(\pi_{a})>\omega(\pi_{b}) .

Let P P be a program and Φ mal \Phi_{\text{mal}} be a malicious behavior specification. If CogniCrypt reports P P as malicious, then there exists an execution trace τ ∈ 𝒯 ⁡ ( P ) \tau\in\mathcal{T}(P) and a formula φ i ∈ Φ mal \varphi_{i}\in\Phi_{\text{mal}} such that τ ⊧ φ i \tau\models\varphi_{i} .

CogniCrypt reports P P as malicious only when the vulnerability classifier 𝒞 vuln \mathcal{C}_{\text{vuln}} returns MALICIOUS for some path constraint π ∗ \pi^{*} generated by the concolic engine. By construction:

Step 1 (Path Feasibility): The concolic engine maintains the invariant that every generated path constraint π \pi is satisfiable, i.e., ∃ 𝐜 ∈ ℐ : 𝐜 ⊧ π \exists\mathbf{c}\in\mathcal{I}:\mathbf{c}\models\pi . This is enforced by the Z3 SMT solver check at line 12 of Algorithm 1 . Therefore, π ∗ \pi^{*} corresponds to a feasible execution trace τ ∗ ∈ 𝒯 ⁡ ( P ) \tau^{*}\in\mathcal{T}(P) .

Step 2 (Classifier Correctness): The vulnerability classifier 𝒞 vuln \mathcal{C}_{\text{vuln}} is trained with a loss function that penalizes false positives with weight w FP = 5.0 w_{\text{FP}}=5.0 (Section 5 ). Under the assumption that the training data is representative of the threat model Φ mal \Phi_{\text{mal}} , the classifier’s positive predictions correspond to traces satisfying some φ i ∈ Φ mal \varphi_{i}\in\Phi_{\text{mal}} with probability ≥ 1 − ϵ \geq 1-\epsilon , where ϵ \epsilon is the empirically measured false positive rate.

Step 3 (Trace-Specification Correspondence): The feature extraction function Extract ​ ( π ∗ , τ ∗ ) \text{Extract}(\pi^{*},\tau^{*}) (Algorithm 2 , line 3) maps the path constraint and its associated trace to a feature vector that encodes the behavioral semantics relevant to Φ mal \Phi_{\text{mal}} . The classifier’s decision boundary partitions the feature space into regions corresponding to the disjuncts of Φ mal \Phi_{\text{mal}} .

Therefore, if CogniCrypt reports P P as malicious, there exists τ ∗ ∈ 𝒯 ⁡ ( P ) \tau^{*}\in\mathcal{T}(P) such that τ ∗ ⊧ φ i \tau^{*}\models\varphi_{i} for some φ i ∈ Φ mal \varphi_{i}\in\Phi_{\text{mal}} , up to the classifier’s error rate ϵ \epsilon . ∎

Let P P be a program, Φ mal \Phi_{\text{mal}} be a malicious behavior specification, and B ∈ ℕ B\in\mathbb{N} be an exploration budget (maximum number of paths). If there exists a malicious trace τ ∗ ∈ 𝒯 ⁡ ( P ) \tau^{*}\in\mathcal{T}(P) with τ ∗ ⊧ φ i \tau^{*}\models\varphi_{i} for some φ i ∈ Φ mal \varphi_{i}\in\Phi_{\text{mal}} , and the corresponding path constraint π ∗ \pi^{*} is within the top- B B paths ranked by ω \omega , then CogniCrypt will detect τ ∗ \tau^{*} .

Step 1 (Exploration Guarantee): The LLM-guided exploration strategy explores paths in decreasing order of ω ⁡ ( π ) \omega(\pi) . Since π ∗ \pi^{*} is within the top- B B paths by assumption, it will be explored within the budget B B .

Step 2 (Detection Guarantee): Once π ∗ \pi^{*} is explored, the concolic engine generates the concrete trace τ ∗ \tau^{*} and the symbolic trace τ ^ ∗ \hat{\tau}^{*} . The vulnerability classifier processes ( π ∗ , τ ∗ ) (\pi^{*},\tau^{*}) and, under the assumption that the classifier has recall ≥ 1 − δ \geq 1-\delta for the malware class corresponding to φ i \varphi_{i} , it will correctly classify π ∗ \pi^{*} as malicious with probability ≥ 1 − δ \geq 1-\delta .

Step 3 (Budget Sufficiency): The LLM’s priority function ω \omega is designed to assign high scores to paths exhibiting patterns correlated with Φ mal \Phi_{\text{mal}} . Empirically (Section 6 ), we demonstrate that ω \omega ranks malicious paths within the top 5% of all paths for 96.8% of malware samples, ensuring that moderate budgets B B suffice for detection. ∎

The path priority function ω \omega is monotone with respect to the path constraint lattice: if π a ⊑ π b \pi_{a}\sqsubseteq\pi_{b} (i.e., π b \pi_{b} is a refinement of π a \pi_{a} ), then ω ⁡ ( π a ) ≤ ω ⁡ ( π b ) + ϵ LLM \omega(\pi_{a})\leq\omega(\pi_{b})+\epsilon_{\text{LLM}} , where ϵ LLM \epsilon_{\text{LLM}} is a bounded approximation error of the LLM.

If π b ⊧ π a \pi_{b}\models\pi_{a} , then π b \pi_{b} constrains the execution to a subset of the paths satisfying π a \pi_{a} . A more constrained path carries at least as much information the program’s behavior. The LLM, having been trained on path-behavior correlations, assigns non-decreasing scores to more informative (more constrained) paths, up to its approximation error ϵ LLM \epsilon_{\text{LLM}} , which is bounded by the LLM’s generalization error on the validation set. Formally, let f : Π ⁡ ( P ) → ℝ d f:\Pi(P)\rightarrow\mathbb{R}^{d} be the LLM’s internal representation function. By the data processing inequality:

where I ⁡ ( ⋅ , ⋅ ) I(\cdot;\cdot) denotes mutual information. Since ω \omega is a monotone function of mutual information (by the classifier’s calibration), the lemma follows. ∎

Under the assumptions of Theorem 2 and Lemma 1 , the expected number of paths explored before detecting a malicious trace is O ⁡ ( | Π ⁡ ( P ) | ω ⁡ ( π ∗ ) ⋅ | Π ⁡ ( P ) | ) = O ⁡ ( 1 ω ⁡ ( π ∗ ) ) O\left(\frac{|\Pi(P)|}{\omega(\pi^{*})\cdot|\Pi(P)|}\right)=O\left(\frac{1}{\omega(\pi^{*})}\right) , which is inversely proportional to the LLM’s confidence in the malicious path.

We consider an adversary 𝒜 \mathcal{A} with the following capabilities:

𝒜 \mathcal{A} has access to one or more LLMs for code generation;

𝒜 \mathcal{A} can generate polymorphic variants: for any malware m m , 𝒜 \mathcal{A} can produce m ′ ≠ m m^{\prime}\neq m such that Behavior ​ ( m ) ≡ Behavior ​ ( m ′ ) \text{Behavior}(m)\equiv\text{Behavior}(m^{\prime}) but Syntax ​ ( m ) ≠ Syntax ​ ( m ′ ) \text{Syntax}(m)\neq\text{Syntax}(m^{\prime}) ;

𝒜 \mathcal{A} can embed trigger conditions: malicious behavior activates only when env ⊧ ψ trigger \texttt{env}\models\psi_{\text{trigger}} for some environmental predicate ψ trigger \psi_{\text{trigger}} ;

𝒜 \mathcal{A} does not have access to CogniCrypt’s internal parameters or training data (black-box assumption).

This section presents the three core algorithms of CogniCrypt in detail.

This section provides a comprehensive description of the CogniCrypt prototype implementation, with sufficient detail to enable full reproducibility.

CogniCrypt is implemented as a modular Python application comprising four principal components: (1) the Concolic Execution Engine, (2) the LLM Path Prioritizer, (3) the Vulnerability Classifier, and (4) the RL Feedback Module. The components communicate via a shared message bus implemented using ZeroMQ (version 4.3.5). Figure 1 illustrates the system architecture.

The concolic execution engine is built on angr version 9.2.100 with the following configuration:

The engine uses Z3 (version 4.12.6) as the backend SMT solver, accessed through angr’s Claripy abstraction layer. We configure Z3 with a per-query timeout of 30 seconds and enable incremental solving for efficiency:

The LLM path prioritizer supports multiple LLM backends through a unified interface. We implement adapters for five LLMs:

The vulnerability classifier is a custom transformer encoder with the following architecture:

The reinforcement learning feedback module uses Proximal Policy Optimization (PPO) [ 25 ] to refine the LLM’s path prioritization policy:

The complete installation can be performed via:

We evaluate CogniCrypt on four benchmark datasets:

The AI-Gen-Malware dataset was constructed by prompting GPT-4, Claude 3, and LLaMA 3 to generate malicious code across 10 categories: trojans, ransomware, spyware, worms, rootkits, backdoors, adware, cryptominers, bots, and polymorphic self-modifying code. Each sample was compiled into a PE binary and verified for malicious functionality in an isolated sandbox environment.

We compare CogniCrypt against the following baselines:

We report Accuracy, Precision, Recall, F1-Score, and Area Under the ROC Curve (AUC-ROC). All experiments use 5-fold cross-validation, and we report mean ± \pm standard deviation.

CogniCrypt achieves the highest performance across all metrics on both datasets. The performance gap is particularly striking on the AI-Gen-Malware dataset, where CogniCrypt outperforms the best baseline (angr-only) by 19.3 percentage points in accuracy and the best ML baseline (MalConv) by 25.1 percentage points. This demonstrates the critical importance of combining concolic execution with LLM-guided analysis for detecting AI-generated threats.

GPT-4 achieves the best detection performance, while Gemini 1.5 Pro offers the best throughput. LLaMA 3 70B and Mixtral 8x22B provide cost-effective alternatives for deployment scenarios where API costs are a concern. All LLMs significantly outperform the no-LLM baseline (angr-only), confirming the value of LLM-guided path prioritization.

A key advantage of CogniCrypt is the efficiency of its LLM-guided path exploration. Figure 4 compares the malicious code coverage achieved by different exploration strategies as a function of the number of paths explored.

To understand the contribution of each component, we conduct an ablation study on the AI-Gen-Malware dataset:

The ablation study reveals that the concolic execution engine is the most critical component (removing it causes a 15.4 pp drop), followed by the LLM prioritizer (9.2 pp drop) and the transformer classifier (6.3 pp drop). The RL feedback loop provides a modest but consistent improvement of 1.7 pp.

To illustrate CogniCrypt’s capabilities, we present a case study involving a polymorphic ransomware sample generated by GPT-4. The sample employs several evasion techniques: (1) environment-aware activation (checks for sandbox indicators before executing), (2) polymorphic encryption routine (generates a unique encryption key and routine at each execution), and (3) anti-debugging measures (detects debugger presence via timing checks).

CogniCrypt’s concolic engine identified 847 unique execution paths in the sample. The LLM prioritizer ranked the path containing the ransomware payload activation as the 3rd highest priority (out of 847), enabling rapid detection. The path constraint for the malicious path was:

The vulnerability classifier assigned a maliciousness score of 0.987 to this path, correctly identifying the sample as ransomware. In contrast, ClamAV and YARA failed to detect the sample due to its polymorphic nature, and MalConv misclassified it as benign due to the obfuscated byte patterns.

This paper introduced CogniCrypt, a novel framework for detecting zero-day AI-generated malware through the synergistic combination of concolic execution, LLM-guided path prioritization, and deep-learning-based vulnerability classification. We established a rigorous theoretical foundation, including a first-order temporal logic for specifying malicious behavior and proofs of soundness and relative completeness, assuming classifier correctness. Our experimental evaluation on four benchmark datasets demonstrated that CogniCrypt significantly outperforms existing detection methods, achieving 97.5% accuracy on AI-generated malware,a 19.3–52.2 percentage point improvement over baselines.

The key insight underlying CogniCrypt is that LLMs, having been trained on vast code corpora, possess an implicit understanding of suspicious program behavior that can be leveraged to guide concolic execution toward malicious paths. This synergy resolves the path explosion problem that has historically limited the scalability of symbolic execution for malware analysis.

Future Work. Several promising directions remain: (1) extending CogniCrypt to analyze Android APKs and IoT firmware; (2) incorporating adversarial training to improve robustness against evasion-aware AI malware generators; (3) exploring federated learning approaches to enable collaborative model training across organizations without sharing sensitive malware samples; and (4) integrating formal verification techniques to provide stronger guarantees on the absence of false negatives. Having established the theoretical foundations of our approach and its accuracy in an experimental setting, we intend to conduct further evaluations in future work. First, we plan to conduct studies on the scalability and deployment feasibility of our approach, given LLMs’ substantial hardware requirements. Second, we will evaluate the approach against additional baselines, including recent transformer-based PE models and hybrid detection systems.

Reproducibility. The CogniCrypt prototype, including all source code, trained models, and the AI-Gen-Malware dataset, will be made available upon publication at .