2406.10279v3

The reliance of popular programming languages such as Python and JavaScript on centralized package repositories and open-source software, combined with the emergence of code-generating Large Language Models (LLMs), has created a new type of threat to the software supply chain: package hallucinations . These hallucinations, which arise from fact-conflicting errors when generating code using LLMs, represent a novel form of package confusion attack that poses a critical threat to the integrity of the software supply chain. This paper conducts a rigorous and comprehensive evaluation of package hallucinations across different programming languages, settings, and parameters, exploring how a diverse set of models and configurations affect the likelihood of generating erroneous package recommendations and identifying the root causes of this phenomenon. Using 16 popular LLMs for code generation and two unique prompt datasets, we generate 576,000 code samples in two programming languages that we analyze for package hallucinations. Our findings reveal that that the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models, including a staggering 205,474 unique examples of hallucinated package names, further underscoring the severity and pervasiveness of this threat. To overcome this problem, we implement several hallucination mitigation strategies and show that they are able to significantly reduce the number of package hallucinations while maintaining code quality. Our experiments and findings highlight package hallucinations as a persistent and systemic phenomenon while using state-of-the-art LLMs for code generation, and a significant challenge which deserves the research community’s urgent attention.

Recent advances in generative AI, powered by Large Language Models (LLMs) like GPT-4 [ 1 ] and LlaMA [ 60 ] , have revolutionized AI capabilities across modalities, excelling in a wide range of tasks such as image synthesis, text generation, and natural language understanding. One such application is code generation, which is typically accomplished by first training or fine-tuning an LLM using vast amounts of programming data found on online repositories (e.g., GitHub), technical forums, and documentation. Both commercial/black-box (e.g., GPT-4 [ 1 ] , Claude [ 3 ] ) and open-source (e.g., CodeLlama [ 53 ] , DeepSeek Coder [ 14 ] ) varieties of such code-generating LLMs are readily available and are extensively used by both novice and expert programmers in their coding workflows to increase productivity. Recent studies indicate that up to 97% of the developers are using generative AI to some degree and that approximately 30% of code written today is AI-generated, reflecting significant perceived gains in efficiency and convenience [ 54 , 36 ] .

One critical shortcoming of LLMs is a phenomenon referred to as hallucination . Hallucinations are outputs produced by LLMs that are factually incorrect, nonsensical, or completely unrelated to the input task. Hallucinations present a significant obstacle to the effective and safe deployment of LLMs in public-facing applications due to their potential to generate inaccurate or misleading information. As a result, there has been increased efforts to research the detection and mitigation of hallucinations in LLMs [ 17 , 22 ] . However, most existing research has focused only on hallucinations in classical natural language generation and prediction tasks such as machine translation, summarization, and conversational AI [ 19 , 45 , 33 , 7 ] . The occurrence and impact of hallucinations during code generation, particularly regarding the type of hallucinated content and its implications for code security, are still in the nascent stages of research. Recently, Liu et al. [ 39 ] have shown that popular LLMs (e.g., ChatGPT, CodeRL, and CodeGen) significantly hallucinate during code generation and have established a taxonomy of hallucinations in LLM-generated code.

In this work, we focus on a specific type of hallucination during code generation called package hallucination . Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations, especially if they are repeated, by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality. As other unsuspecting and trusting LLM users are subsequently recommended the same fictitious package in their generated code, they end up downloading the adversary-created malicious package, resulting in a successful compromise. This compromise can then spread through an entire codebase or software dependency chain, infecting any code that relies on the malicious package. This is a variation of the classical package confusion attack that has been enabled by code-generating LLMs.

Package confusion attacks, through techniques such as typosquatting (i.e., creating packages with names similar to popular ones to deceive users) and name similarity, have been a long-standing issue in the open-source software community [ 47 , 55 , 27 ] . Package hallucinations by code-generating LLMs threaten to exacerbate the problem by exposing an additional threat surface for such attacks. Trivial cross-referencing methods (i.e., comparing a generated package name with a list of known packages) are ineffective for detecting a package hallucination attack, as an adversary may already have published the hallucinated package with malicious code. Open-source repositories make no guarantee the safety of hosted content; the mere presence of a package in an open-source repository does not confirm its credibility. A recent blog post [ 29 ] suggests that LLMs are prone to package hallucinations and provides a first approximation of their prevalence, but the extent to which this phenomenon occurs in state-of-the-art (SOTA) commercial and open-source LLMs, the nature of these hallucinations, and the effectiveness of potential mitigation measures have not been thoroughly investigated before.

In this paper, we conduct the first systematic study of the frequency and nature of package hallucinations across a variety of code-generating LLMs, operating under a diverse set of model settings and parameters. We specifically make the following novel contributions:

Characterizing the prevalence of package hallucinations by code-generating LLMs and related functional attributes : We first comprehensively analyze the prevalence of package hallucinations in Python and JavaScript code generated by popular commercial and open-source LLMs. We also examine and characterize commonly observed LLM behaviors related to package hallucinations, including hallucination repetition, output verbosity, and the ability of these models to detect their own hallucinations.

Analyzing the effect of fine-grained changes to model settings on package hallucinations: We further study how specific model settings, such as training data recency, model temperature, and decoding strategies affect the occurrence and nature of package hallucinations.

Characterizing common traits of the generated hallucinated packages : We carefully study several key properties of the hallucinated packages, such as their semantic similarity to popular packages, their propensity of occurring across different models, and the influence of packages that were recently removed/deleted (from the corresponding repositories) on the hallucination rate, among others.

Testing of mitigation strategies : We propose and comprehensively evaluate several techniques to effectively mitigate package hallucinations in LLM-generated code while maintaining the ability to produce effective code.

Publicly-accessible datasets for advancing research: We make publicly available two novel datasets (one Python and one JavaScript) of 19,500 coding prompts for a wide range of coding tasks and 586,000 generated coding samples for fine-tuning/analysis. 1 1 1 All code and datasets can be found at: or

In this section, we provide a brief background on open-source software security, code-generating LLMs, and the issue of hallucinations in LLMs.

Package Confusion Attacks in Open-source Software Repositories. Modern software development has seen an increased reliance on open-source software packages and libraries that are publicly-available on centralized repositories. Many modern programming languages now rely on such centralized package repositories, with PyPI [ 9 ] (for Python) and npm [ 48 ] (for JavaScript) being the two most popular repositories. The open nature of these repositories, where anyone can upload new code packages/libraries, makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone [ 54 ] .

Once a malicious package is uploaded, adversaries employ various techniques to trick users into downloading it, thereby integrating it into their codebases and dependency chains. These attacks often involve deliberately naming malicious packages to mimic legitimate ones, a tactic known as a package confusion attack [ 47 ] . Package confusion attacks can be broadly categorized into typosquatting , combosquatting , brandjacking , and similarity attacks [ 28 ] , and are distinct from other types of software supply chain attacks such as corrupting legitimate packages or developing unique malicious packages from scratch as part of a long-term campaign. More than 1,200 package confusion attacks have been documented in the last six years [ 47 ] , including the notable PyTorch compromise [ 55 ] and the Lazarus Group campaign [ 27 ] .

Packages/libraries often rely on other packages to function, thus creating extensive dependency trees . Infecting a single package in this dependency chain can be sufficient to compromise an entire software product or ecosystem [ 47 , 26 ] . Public OSS repositories such as PyPI and npm have implemented various measures, including two-factor authentication, namespace protection, and software signing to mitigate the distribution of malicious packages [ 61 , 66 ] . However, it remains unclear whether these repositories utilize any scan-based techniques for detecting malicious code, and they often do not disclose the full list of removed packages.

Automated Code Generation using LLMs. Modern LLMs continue to demonstrate advanced source-code generation capabilities, with success rates in correctly answering coding prompts surging from 25% in June 2021 to 96% by April 2024 [ 6 ] . With the increasing use of these models for software development, concerns are increasing the likelihood of producing insecure or incorrect code that could create vulnerabilities in deployed applications. Early versions of code-generating LLMs were found to generate code containing vulnerabilities listed in the MITRE Top-25 Common Weakness Enumeration (CWE) 40% of the time [ 51 ] . Moreover, recent research has shown that AI-assisted programming not only results in less secure code, but also instills a false sense of security among developers [ 52 ] .

Hallucinations by LLMs. It has been well documented that LLMs can unintentionally produce harmful information [ 41 , 64 ] , be manipulated for malicious purposes [ 15 , 25 ] , expose private information [ 34 ] , and carry inherent biases in their training data [ 11 ] . A related phenomenon is hallucinations, where LLMs generate misleading or entirely fictitious information. These errors take various forms: the model might misinterpret the intended input ( input-conflicting hallucination ), produce inconsistencies with output ( context-conflicting hallucination ), or contradict established facts ( fact-conflicting hallucination ) [ 67 ] . Hallucinations can arise from three main root causes: (i) data , (ii) training , and (iii) inference [ 18 ] . Data-related hallucinations occur when the source data itself is flawed with misinformation [ 37 ] , bias [ 11 ] , or incomplete records [ 50 ] . Architecture flaws [ 38 ] or suboptimal training objectives [ 62 ] during training could also result in downstream hallucinations, while inference time issues such as defective coding strategies [ 16 ] and imperfect decoding representations [ 5 , 42 ] are other contributors. The probabilistic nature of LLMs presents a challenge in mitigating hallucinations. This nondeterminism, while it fosters creativity and generates diverse and innovative content, also contributes to the generation of hallucinated content. Balancing creativity with accuracy remains a central challenge in deploying LLMs, underscoring the complexity of developing effective mitigation strategies.

Package Hallucinations and Security Risks. Package hallucinations, a special form of fact-conflicting hallucinations, are instances where LLMs generate fictitious (non-existent) or erroneous package names in the generated source code. As outlined earlier, an adversary can quickly create malicious packages (on the appropriate open-source repository) with the same name as these hallucinated packages, thus effecting a very simple, yet effective, form of package confusion attack. Unsuspecting users, who trust the LLM output, may not scrutinize the validity of these hallucinated packages in the generated code and could inadvertently include these malicious packages in their codebase [ 52 ] . This resulting insecure open-source code also has the potential of being included in the dependency chain of other packages and code, leading to a cascading effect where vulnerabilities are propagated across numerous codebases. The simplicity and scale of such LLM-enabled package confusion attacks highlight the critical need for quantifying this existing risk, understanding the nature of this unique type of hallucination, and developing effective mitigation techniques that maintain the utility of the code generated by the LLMs. This is precisely what we aim to accomplish in this work.

The possibility of code-generating models recommending malicious or typosquatted packages was first suggested in 2021 as tools such as GPT-3 and Codex were released as viable code generation platforms [ 4 ] . At the time, the risk of these tools suggesting vulnerable, malicious, or typosquatted packages was assessed to be low [ 6 ] . In particular, the related but distinct concept of package hallucinations was not explicitly considered in this initial risk assessment; either because such an attack scenario was not considered at all or because the threat was thought to be negligible. The capabilities of generative AI agents have advanced significantly since that introductory evaluation.

Although a comprehensive study on the prevalence of package hallucinations in LLM-generated code has not been done previously, a recent blog post by Lanyado [ 30 ] outlines the results of some preliminary tests on commercial LLMs such as GPT, Gemini, and Cohere. Their testing confirms the presence of hallucinated packages in the code generated by these LLMs, but their initial estimate of hallucination rate was 5 × \times × higher than our findings, and they do not consider popular open-source LLMs or study possible mitigation approaches. They also fail to systematically characterize the generated hallucinated packages and model properties that have an impact on hallucinations.

In contrast, we conduct a rigorous and comprehensive evaluation across a broader range of models, including the first analysis of package hallucinations in open-source models of any kind, at a scale that has not been previously done. To this end, we provide thorough testing with a larger custom dataset covering two programming languages (namely, Python and Javascript), followed by a detailed analysis of the significant characteristics of this phenomenon.

Adversary Model and Assumptions. We assume an adversary who wants to execute a package confusion attack by leveraging package hallucinations in the code generated by closed-source and open-source code-generating LLMs (see Figure 1 ). The target of the adversary are users who employ such LLMs for generating code. Here we assume that the LLM generates code that requires additional packages for the user to install, which the user does without sufficient verification of the recommended packages. In other words, the target users fully trust the LLMs to include only valid package names in the generated code. We assume that the adversary has access to the same set of LLMs for code generation as the target users, and is unable to modify or manipulate the model and model parameters of these LLMs (e.g., via retraining or fine-tuning) before they are used by the victims. The adversary is able to determine a list of hallucinated packages generated by these LLMs (for example, by cross-referencing the package repository), and then is able to realize a package confusion attack by creating a package of the same name on the corresponding package repositories. These newly created (and now publicly available) packages by the adversary could contain malicious code or functionality. Research has shown that installing open source Python or JavaScript packages allows the execution of arbitrary code by an attacker [ 49 ] . work [ 30 ] has also established the viability of such an attack by publishing a hallucinated package to an open source repository and demonstrating that the package is actively downloaded and was incorporated into the dependency chains of other packages/code.

Research Questions. We now organize our investigation into the following five broad Research Questions (RQ).

RQ1: How prevalent are package hallucinations while generating Python and JavaScript code using LLMs? Our aim here is to thoroughly examine how often package hallucinations occur with both widely-used commercial and open-source LLMs when they generate Python and JavaScript code across various programming tasks.

RQ2: How are package hallucinations impacted by select model settings? Here our goal is to comprehensively analyze how training data and decoding strategies impact the package hallucinations produced by these code generation LLMs.

RQ3: What are the commonly observed model behaviors related to package hallucinations? This RQ will exhaustively study model behaviors such as hallucination repetition by a single LLM (hallucination persistence) and across multiple LLMs (cross-model hallucinations), output verbosity, and the ability of LLMs to detect their own hallucinations (after generation).

RQ4: What are some of the defining properties/attributes of the observed package hallucinations? The goal of this RQ is to analyze the properties of the hallucinated packages such as semantic similarity between hallucinated and popular packages, number of cross-language hallucinations (i.e. non-existent packages from the language requested but valid packages in another programming language), and the number of generated packages that were recently removed from the source repositories.

RQ5: Is it possible to effectively mitigate package hallucinations using best practices in the literature and knowledge gained from earlier results? Through this RQ, we will investigate if code-generating LLMs can be designed to reduce hallucinations with minimal compromise to code quality. In this direction, we will study if techniques such as retrieval augmented generation (RAG) [ 32 ] , self-detected feedback, decoding strategies, and supervised fine-tuning [ 57 ] are effective package hallucination reduction strategies.

To address the RQs outlined above, we design several experiments to repeatedly prompt LLMs to generate code and then analyze the generated code. Our experimentation pipeline consists of three distinct phases: (i) prompt dataset generation , (ii) code generation , and (iii) hallucination detection , each of which is described .

The experiments are designed to exhaustively test each LLM through a complete range of coding tasks. Existing benchmark datasets of coding prompts contain only a limited number of prompts (e.g. only 164 prompts for both EvalPlus [ 40 ] and HumanEval [ 6 ] ) and lack diversity. Therefore, we create a new code prompt dataset for our experiment that contains both breadth and depth in terms of overall number of prompts and range of topics. Our goal was to develop a dataset that accurately and comprehensively represents the coding tasks commonly requested by everyday users. To accomplish this, we employ two distinct approaches, as described below.

Stack Overflow Dataset. To model the input prompts around real programmer questions, our first prompt dataset was created using Stack Overflow [ 56 ] questions across relevant programming topics and subject areas. Stack Overflow is a popular online question-and-answer service for software programmers and developers. To capture a wide range of topics, we utilize the “tag” feature of Stack Overflow, which allows users to label posts according to a subject matter. We included any tag that had more than 5,000 questions and was also relevant to Python or JavaScript (the two programming languages that we focus on in this work, as detailed in Section 4.2 ). For each of the 240 manually selected tags that met this criterion (a full list of tags can be found with the paper artifacts 1 ), we extracted the 20 most upvoted questions, resulting in 4,800 prompts (i.e., 4,800 prompts for Python and 4,800 prompts for JavaScript).

As more recent data is less likely to be included in the pre-training data of LLMs, we are also interested in investigating the temporal correlation between data recency (i.e., how recently the question was asked on Stack Overflow) and model hallucination rate. To enable such an analysis, we ran two queries on Stack Overflow; one that captured only the most popular questions in the selected tags from 2023 and another that captured the most popular questions for all years prior to 2023. By including the two different ranges of time, we effectively doubled the original number of prompts, for a total of 9,600 for each of the two languages.

Not all questions asked on Stack Overflow may involve coding or require code to answer the question. Rather than attempting to filter out such prompts during the code generation phase, which is non-trivial and error-prone, the LLM is asked to answer the question and only provide code if necessary. In the end, this may result in a slightly lesser number of usable LLM-generated code samples but is more realistic as LLMs are expected to accommodate imperfect user inputs.

LLM-generated Dataset. As a majority of the programming tasks require some library/package, our idea was to use the package repositories themselves as a good representation of the full spectrum of coding topics. Our goal was to represent as many code generation tasks as possible in one comprehensive dataset. We take the 5,000 most popular Python and JavaScript packages (based on the number of downloads) and scrape the official package description as listed on PyPI and npm, respectively. These descriptions are then individually inputted to the Llama-2 70B model with instructions to generate a coding prompt based on the package description (the exact prompt available in Figure 13 ). This process generated roughly 4,800 prompts for Python and JavaScript each, resulting in two datasets of approximately the same size (some packages with no description or descriptions in a non-English language were discarded). Similarly to the Stack Overflow dataset, we doubled the LLM-generated dataset for temporal analysis by dividing it into two segments: the packages most downloaded in 2023 and the packages most downloaded prior to 2023. When a package appears in both sets, we remove the package from the latter set to ensure that there is no overlap. Removing duplicates from the latter dataset guarantees that the remaining packages will be those that have increased in popularity during the last year, capturing the desired signal. A truncated list of the LLM-generated dataset can be found in Appendix A .

For brevity, the two temporally distinct datasets, one from the past year and one from before 2023, will be referred to as the ’recent’ and ’all-time’ datasets for the remainder of this paper.

Model Selection. For our experiments, we chose the models that were the highest ranked on the EvalPlus leaderboard (as of January 20, 2024) [ 40 ] . During the creation of our model list, we ignored the fine-tuned versions that were ranked below their corresponding foundational models and only selected one fine-tuned version of the same foundational model of the same parameter size [ 40 ] . EvalPlus maintains a ranking of the top performing LLMs for code correctness according to a rigorous code synthesis evaluation framework. Our goal was to include a mix of top-performing base models and a few of the best-performing fine-tuned variants. We also included the GPT series of models (GPT-3.5, GPT-4, and GPT-4 Turbo) in our experiments, which currently hold the top rankings on the leaderboard. GPT models are widely considered as SOTA in terms of code generation models at the time of writing and add value to our experiments as representative commercial models. The models were not modified or altered in any way prior to testing; they are strictly “off-the-shelf.” Table 1 provides a complete list of the models that we tested in our experiments.

Language Selection. In our experiments, we focus on two of the most popular programming languages, JavaScript and Python. These languages were chosen due to their overall popularity (#1 and #2 according to the GitHub 2023 Octoverse report [ 12 ] ) and their dependence on open-source repositories for package management. Other popular programming languages like Java, C, or C++ do not rely on a centralized open-source repository, as Python and JavaScript do, which is a key component of this vulnerability. The open-source package repositories for these languages, npm and PyPI, represent ecosystems of 5.1 million and 573 thousand packages, respectively [ 58 ] . Of the 16 total models tested (see Table 1 ), 14 were tested for both Python and JavaScript, while two fine-tuned Python-specific models, WizardCoder-Python and CodeLlama-Python, were only tested for Python.

Testing Environment. All open-source models were tested using the Hugging Face transformers package and quantized versions of the models, which reduces parameter precision to boost inference speed and lower memory use without significantly impacting performance. Specifically, the GPTQ quantization method was used, which utilizes a one-shot weight quantization method based on approximate second-order information that has a negligible effect on the accuracy of models, making it an ideal choice [ 10 ] . Additionally, quantized models better simulate the performance that a typical user can expect when running models on commercial grade hardware, making them more accessible and practical for everyday use.

For testing uniformity, we use the same parameters and quantization precision for all open-source models, which are summarized in Appendix C , along with the computing environment used. To generate code for our analysis, we query each LLM ( Table 1 ) with prompts from the two datasets along with a system message which contains specific instructions regarding the task and output format. An overview of the process, including the system messages used during each step, is detailed in Appendix B . The experiment generates 19,200 code samples per model (16 Python tests + 14 JavaScript tests * 19,200 = 576,000 total code samples), which are further analyzed to determine which packages are required to execute the generated code.

To detect hallucinated packages, we first need to extract package names from the LLM output or the generated code sample, which is non-trivial. Simply parsing the code for “import” or “require” is not useful, as the arguments in those statements refer to modules and not packages. There is no way to definitively determine the required packages from a code snippet alone. A detailed explanation of this problem can be found in Appendix G . To solve this problem, we employ the following three heuristics to determine/identify package names in the generated code:

Heuristic 1. As part of our first heuristic, we parse the generated Python and JavaScript code for “ pip install ” and “ npm install ” commands, respectively. These commands look for the specified package in the PyPI/npm repository, resolve its dependencies, and install everything in the current Python/JavaScript environment to ensure that future module requests will work. This is the most straightforward heuristic for detecting package names (and thus hallucinations), as it involves explicit commands from the code generation model for package download/installation. This is significant because if the referenced hallucinated package was indeed used by an adversary to execute a package confusion attack, it could immediately trigger download/install of the malicious code in the package. Note that we did not directly ask the model to provide these commands, but allowed them to occur naturally during the generation process. As such, we observed that these instances (“ pip install ” and “ npm install ”) occur for 7 % percent 7 7\% 7 % of the total output.

Heuristic 2. For the second heuristic, each generated code sample is used as input to the same model that generated it. The model is then prompted for a list of packages that would be required to run the given code. Our intuition is to mimic an actual user/developer who is using LLMs for code generation. If the user gets an error due to an uninstalled package when attempting to execute the generated code, they could query the model for the correct package to install. We wanted to replicate this intuitive process to identify the package names required by the generated code.

Heuristic 3. As the third heuristic, we reuse the original prompt used to generate the code sample as an input to the model and ask the model to output package names that would be required to accomplish this coding task. Similarly to the heuristic, this process of extracting package names simulates another approach users would take to obtain package names from the model that generated the code, if the required packages were not mentioned in explicit “ pip install ” and “ npm install ” commands.

Once each model provides specific package names (through the three heuristics outlined above), we simply compare each package name to a master list of package names acquired from PyPI and npm, respectively (each list is as of 10 January, 2024). If a package name is not on the master list, it is considered a hallucination. We acknowledge the possibility that the master list of packages obtained from the package repositories has already been contaminated with malicious hallucinated packages. It is not possible to guarantee that the master list actually represents the ground truth of valid packages; however, the presence of hallucinated packages already in the master list would actually produce fewer hallucinations, and therefore our results represent a lower bound of hallucination rate.

In this section, we present the results of our experimental analysis related to RQ1 − - - RQ4. After using both Python and JavaScript for RQ1, for RQs 2 through 4, we focus our analysis only on the Python programming language, a subset of the original models tested, and randomly sampled subsets of our original datasets. Given the consistent results that we were able to obtain across both languages for RQ1, we believe that this narrowed scope of discussion for RQs 2 through 4 should not compromise the generalizability of the conclusions and would allow for a deeper analysis of package hallucinations in a controlled setting. We selected GPT-4 Turbo, GPT-3.5, CodeLlama 7B, and DeepSeek 6.7B for the in-depth analysis of RQs 2-4, representing the best-performing and most popular open-source models.

In our first experiment, our goal was to quantify the prevalence of package hallucinations across different models by generating and analyzing a large number of code samples. We conducted 30 tests (using 16 models for Python and 14 models for JavaScript, as described in Table 1 ) producing a combined 576,000 code samples using both the Stack Overflow and LLM-generated datasets ( Section 4.1 ). Each code sample was evaluated for hallucinations according to the heuristics defined in Section 4.3 , which include parsing the generated code and prompting the model for packages twice per code sample, for a total of 1,152,000 package prompts across all tests. To measure LLMs’ propensity to produce hallucinated packages during code generation, we use the package hallucination rate metric, which can be expressed as a simple ratio of the number of hallucinated packages to the total number of recommended packages. The total hallucination rates for each evaluated model are presented in Figure 2 . More fine-grained results on hallucination rates for all models tested, covering both Python and JavaScript, are presented in Appendix E .

These 30 tests generated a total of 2.23 million packages in response to our prompts, of which 440,445 (19.7%) were determined to be hallucinations, including 205,474 unique non-existent packages (i.e. packages that do not exist in PyPI or npm repositories and were distinct entries in the hallucination count, irrespective of their multiple occurrences). Our results for GPT-3.5 (5.76%) and GPT-4 (4.05%) differ significantly from work on package hallucinations [ 30 ] , which found hallucination rates 4 − - - 6 times higher (24.2% and 22.2%, respectively) for those specific models. GPT series models were found to be 4 times less likely to generate hallucinated packages compared to open-source models, with a hallucination rate of 5.2% compared to 21.7%. GPT-4 Turbo resulted in the lowest overall hallucination rate at 3.59%, while DeepSeek 1B had the best hallucination rate among open-source models at 13.63%. Python code resulted in fewer hallucinations than JavaScript (15.8% on average compared to 21.3% for JavaScript). Despite the difference in hallucination rate between the two languages, there is a linear relationship between the results (as shown in Figure 14 in the appendix), demonstrating that the propensity of a model to hallucinate is positively correlated between programming languages. The above results provide strong evidence that package hallucinations are a pervasive issue across all code-generating LLMs.

RQ1 Summary: Package hallucinations were found to be pervasive phenomenon across all 16 models tested. Commercial models hallucinated 4 × \times × less compared to open-source models. Python code resulted in a lower hallucination rate compared to JavaScript.

Effect of Temperature Settings. The temperature setting in a LLM is used to adjust the randomness of the generated responses, where a lower temperature results in more predictable and deterministic outputs, while a higher temperature increases creativity and diversity in the responses ( Section F.1 ). We varied this setting for each model between the minimum and maximum allowed values and observed the change in hallucination rate (the maximum temperature for the GPT series models is limited to 2, while the open-source models can be set to 5). All models exhibited a clear increase in hallucination rate as temperature value increases , with the effect becoming severe at maximum values. The OpenAI models, as shown in Figure 3 , showed only a slight increase in hallucination rate between temperatures 0 and 1, which then increased sharply between 1 and 2. In particular, GPT-4 resulted in a hallucination rate (8.9%) nearly 4 times lower than GPT 3.5 (31.8%) at its maximum temperature. At the highest temperature values, open-source models start to generate more hallucinated packages than valid packages. Most LLMs operate at a default temperature in the range of 0.7 to 1, however, our results indicate that a lower temperature value can reduce package hallucinations, with the optimal value varying per model. Lower temperature also yields more deterministic responses, presenting a trade-off between risk of hallucination and creativity. Therefore, selecting the appropriate temperature value is not a straightforward decision.

Effect of Decoding Strategies. , we adjusted several decoding parameters (top- p 𝑝 p italic_p , top- k 𝑘 k italic_k , and min- p 𝑝 p italic_p values) to reduce the chances of a low probability token being selected as a potential package, with the intuition that lower probability tokens correspond to higher probabilities of hallucination in this context. The following is a summary of the parameters and values we modified.

Top- p 𝑝 p italic_p (0.4, 0.6 , 0.8): Tokens with probabilities adding up to less than this number are discarded.

Top- k 𝑘 k italic_k (5, 10 , 15): Select only the top- k 𝑘 k italic_k most likely tokens.

Min- p 𝑝 p italic_p (0.1, 0.2 , 0.3): Tokens with probability smaller than (min- p 𝑝 p italic_p * probability of most likely token) are discarded.

We evaluated each listed value in isolation, followed by a combined evaluation of the values highlighted in bold, resulting in a total of 10 tests. Note that top- k 𝑘 k italic_k and min- p 𝑝 p italic_p were only tested for DeepSeek and CodeLlama, as these values are not modifiable through the OpenAI API. Varying the decoding values induced a slight increase (1.16% on average) in the hallucination rate for the four models across all values tested. As we will expand on in RQ3, package hallucinations are often persistently repeated across many iterations.

This suggests that greedy decoding strategies, which prioritize the most probable tokens (i.e., the most probable token is always selected), would still generate fictitious packages. This differs from other types of hallucinations, which generally occur when low-probability tokens are sampled. This persistent nature of package hallucinations highlights the inherent complexity of the problem.

Recency of Subject Matter. As described in Section 4.1 , we separated our coding prompts into two temporal datasets to evaluate whether the hallucination rate was correlated with topics/packages that emerged after the model was trained. A lower difference between the rates of recent and all-time prompts indicates better performance in handling questions that fall outside the model’s pre-training data and therefore a more generalizable model. The models we tested were shown to be more likely to generate a package hallucination when responding to prompts that deal with more recent topics. This difference resulted in a 10% higher hallucination rate on average for older data versus more recent data.

Overall, all 16 Python models we evaluated demonstrated a higher hallucination rate when being prompted questions or packages that were popular within the past year ( Figure 4 ). These higher rates are at least partially due to the inherent limitations and training costs of modern LLMs. As noted in the OpenAI GPT-4 technical report [ 1 ] , LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date. Although fine-tuning can enhance specific tasks, it generally does not improve the model’s overall knowledge of the world. The massive cost of training LLMs from scratch, evidenced by the 1,400,000 GPU hours (220 years) required to train the 12 CodeLlama models, makes continuously updating pre-training data impractically expensive [ 53 ] . This cost, along with steadily increasing model sizes and training times, poses a significant barrier to reducing package hallucinations for advanced coding prompts and packages.

RQ2 Summary: Lower temperatures result in the lowest hallucination rate while hallucination rates increase dramatically with temperature values larger than 1. Altering decoding and sampling parameters in the model does not improve hallucination rates. More recent prompting topics lead to a 10% increase in hallucination rates.

Frequency of Repeated Hallucinations. To determine whether hallucinations are random error or repeatable phenomena, this test focuses on the persistence of hallucinations within a model. We randomly sampled 500 prompts that generated package hallucinations during our initial testing and then repeated those queries 10 times per prompt. Of those 10 queries, we recorded how many times the original hallucinated package was regenerated. Our analysis reveals an unexpected dichotomy when repeatedly querying a model with the same prompt that generated a hallucination: 43% of hallucinated packages were repeated in all 10 queries, while 39% did not repeat at all across the 10 queries. This is indicated in Figure 5 , which shows prominent spikes at zero repetitions and at 10 repetitions, respectively, for all models. In addition, 58% of the time, a hallucinated package is repeated more than once in 10 iterations, which shows that a majority of hallucinations are not simply random errors, but a repeatable phenomenon that persists across multiple iterations . This is significant because a persistent hallucination is more valuable for malicious actors looking to exploit this vulnerability and makes the hallucination attack vector a more viable threat.

Verbose Models versus Conservative Models. LLMs operate with inherent randomness and uncertainty. This behavior enables novel and creative output, a desired feature for many NLP tasks but less welcome for code generation, which requires a high degree of accuracy and must adhere to rigid syntax. We define a verbose model as one that operates with higher degree of uncertainty and randomness by generating a greater number of distinct package names while a conservative model generates a lesser number of distinct packages, generally using only the most popular and well-known packages. To this end, we investigated whether verbose models correspond to a higher rate of package hallucinations. Our results ( Figure 6 ) show a correlation between hallucination rate and number of unique packages that were recommended during this experiment (i.e. a more verbose model was associated with a higher hallucination rate). In light of these findings, it is reasonable to suggest that models generating code should adopt a more conservative approach (i.e. limiting package suggestions to a smaller list of well-known packages rather than generating names with uncertainty).

The models with the lowest hallucination rates in our study adhered to a smaller subset of packages when generating code, and these models (e.g., the GPT series) also scored the highest on the EvalPlus [ 40 ] code quality benchmarks. This suggests that improving code quality and reducing hallucinations can potentially be achieved simultaneously without a trade-off.

LLMs’ Ability to Detect Hallucinations. We then evaluated each model’s ability to identify hallucinations versus valid packages, both from its own code generation outputs and those generated by other models. To test this, we conducted two binary classification tests: (i) each model’s ability to detect hallucinated packages from its own generated code and (ii) each model’s ability to detect hallucinated packages from code generated by other models. The names of the valid and hallucinated packages produced by each model were randomly sampled, and each model was asked "Is [package name] a valid Python package?" Identification accuracy was calculated as the ratio of correct identifications to the total number of packages provided.

Figure 7 shows that 3 of the 4 models (GPT 4 Turbo, GPT 3.5, and DeepSeek) proved to be highly adept in detecting their own hallucinations with detection accuracy above 75%. Table 2 displays the recall and precision values for this test, with similarly strong performance across the 3 proficient models. This phenomenon implies that each model’s specific error patterns are detectable by the same mechanisms that generate them, suggesting an inherent self-regulatory capability. The indication that these models have an implicit understanding of their own generative patterns that could be leveraged for self-improvement is an important finding for developing mitigation strategies.

CodeLlama displays unique and interesting behavior during both tests, as it has an overwhelming propensity to label most packages as valid, resulting in a lower accuracy for hallucinated packages.

RQ3 Summary: Package hallucinations are often persistently generated. Models that generate fewer packages when prompted are correlated with a reduced hallucination rate. Several models were able to detect their own hallucinations with greater than 75% accuracy.

Occurrence of the Same Package Hallucination Across Different Models. To analyze the possibility that the same hallucinated packages are generated across different models, we measured how many models generated the same package name given a confirmed package hallucination. Figure 8 shows a clear pattern in which a large majority (81%) of distinctly generated package names were generated by only one model . In other words, the specific package names were usually unique to a single model, where only the most common packages were generated by more than one model. The two populations (valid and hallucinated packages) diverge as the number of models increases, with the number of hallucinated packages decreasing nearly exponentially and the distribution of valid packages becoming more uniform. The finding that valid packages are less dependent on the specific model used for generation is attributable to their frequent appearance in training data and applicability to universal coding problems, leading to their widespread use in a broad range of prompts.

Combining the insights gained during the persistence analysis ( Figure 5 ) leads to a key observation. As previously shown, hallucinations are often persistent (58% are repeated within 10 iterations) within the same model but are not often repeated between models, as 81% of hallucinated packages are generated by only one model. This further reinforces the evidence that while hallucinations are a common phenomenon across various models, the exact nature of these hallucinations is generally model specific. This behavior is particularly surprising given that our testing includes multiple models from the same family (i.e. 3 GPT models, 4 CodeLlama models, and 3 DeepSeek models). These models presumably use the same training data for each version, yet each model generates unique hallucinations that are not found in other models.

Semantic Similarity Between Hallucinated and Popular Valid Packages. In order to analyze the semantic similarity between hallucinated and popular real/valid packages, we measured the average Levenshtein distance of a package to its nearest neighbor (i.e., the closest valid package). Levenshtein distance is a measure of how many insertions, deletions, and substitutions are required for two strings to match [ 31 ] . If the distribution of Levenshtein distances is skewed heavily right, with a peak at or near 0, this would indicate that most hallucinations are very similar to valid package names. In that case, attackers could infer a hallucination target based on more traditional package confusion methods (e.g., typosquatting) rather than analyzing a large volume of model output over time to detect persistent hallucinations that could be used as vessels for malicious code. A higher distance reflects that package hallucinations are more random in nature and difficult to predict, rather than the result of minor grammatical errors.

The results of the Levenshtein distance test, as seen in Figure 9 , suggest that most package hallucinations are not simple off-by-one errors . An off-by-one error in our case refers to a discrepancy between the hallucinated package and its nearest match, involving a difference of 1 to 2 characters, including numbers, letters, or punctuation marks. Our results show that only 13.4% (10,263 of 76,489) have a Levenshtein distance of 1 or 2. Another 37.9% (29,025 of 76,489) of packages registered a score between 3 and 5, which would indicate two words with a common root word or concept that still differ significantly. In particular, 48.6% (37,207 of 76,489) of hallucinations scored 6 or higher, with 20.2% (15,457 of 76,489) of those scoring 10 or higher, indicating two strings that are very different and likely do not any common theme.

The presence of such a large proportion of high Levenshtein values suggests that the majority of hallucinations are not merely trivial typographical errors but are substantively different from existing package names . The observed results provide further evidence that the root cause of hallucinations is likely to be more complex than minor string manipulation, pointing to deeper issues in the model’s generative processes that govern the creation of package names. The long right tail of the distribution in Figure 9 indicates a wide variety of hallucinations spread over a broad range, revealing a diversity in types of errors and reinforcing that the generation of hallucinations is a complex issue not limited to simple character substitutions, additions, and deletions.

Effect of Deleted Packages. To determine whether packages that existed before a model’s pre-training data cut-off date (i.e., the final day of data included in the model’s training set) but were subsequently removed contribute significantly to package hallucinations, we conducted an analysis using package download counts obtained via Google BigQu...