ASTER: Using large language models (LLMs) to generate multilingual and natural unit tests.
One crucial yet time-consuming step in software development is the implementation of automated unit testing. Numerous methods for automating the creation of unit tests have been developed to help developers with this chore. Despite this endeavor, only a small number of programming languages have viable tools available. For this reason, the Automated Test Case Generation, or ASTER, was created by the IBM Research team.
Automated Test Case Generation(ASTER) shows that high-coverage and natural tests can be produced via LLM prompting directed by lightweight program analysis. This method is implemented for Python and Java by ASTER. Automated Test Case Generation(ASTER) can also handle complex scenarios requiring external dependencies, including database transactions or web service calls, to its incorporation of software mocking methods.
Let’s go over the main difficulties developers encounter while creating unit tests and discuss how Automated Test Case Generation(ASTER) can assist in order to completely grasp why believing this work is so crucial.
The challenge of unit test generation
It takes a lot of time. Unit tests enable developers to thoroughly verify the application’s functionality. The tests concentrate on how each unit usually a method, function, or procedure is implemented. Writing maintainable and highly covered tests, however, may be laborious and time-consuming, which lowers developer productivity and speeds up the development of new or upgraded enterprise applications.
Comparison of LLM-assisted ASTER test cases (right) with EvoSuite and CodaMosa tests (left) for naturalness (test names, variable names, and assertions) and mocking.
The “naturalness” of tests produced using traditional methods is lacking. According to earlier research, developers believe that automatically created exams lack “naturalness” qualities, are difficult to read and understand, cover dull passages, and make pointless or ineffective claims. Because they differ from the types of tests that developers write, automatically generated tests are typically not seen as natural and are known to have anti-patterns.
Because developers view the tests produced by these tools as difficult to maintain and are hesitant to include them in regression test suites without a significant amount of rewriting, all of these reasons prevent the practical use of test creation tools. For example a few test instances produced by CodaMosa and EvoSuite. An experienced developer would not create test cases with variable names, test names, and test assertions that lack sense. Indeed, according to a poll of IBM internal developers, developers would typically not include these tests in their regression test suites and do not find them valuable (Figure 1).
LLM-generated tests frequently fail to run or build. Developers can create tests that appear more realistic by using LLMs. Due to their limited access to the application under test, those models are susceptible to hallucinations, which is why the generated tests frequently fail to compile and execute. Developers must frequently correct these tests before using them, which can sometimes require more work than creating the tests from scratch.
Ready-to-use tests for a variety of programming languages are not supported. There are ready-to-use test generating tools for a few major programming languages, including Java, C, and Python, despite decades of research on the subject. It takes a lot of work to expand them to accommodate additional program languages.
Automated Test Case Generation(ASTER): Static-analysis-guided pipeline
ASTER uses four distinct procedures to solve each of these concerns:
- Preprocessing by static analysis: To extract important contextual information, the system thoroughly analyses the application under test using static analysis. In order to create meaningful tests, it is essential to discover method signatures, call hierarchies, and any dependencies.
- LLM-guided test generation: Automated Test Case Generation(ASTER) creates comprehensive LLM prompts by utilising the knowledge gained from static analysis. The models are guided by these prompts to produce unit tests that are semantically rich, syntactically accurate, and consistent with human coding conventions.
- Postprocessing and refinement: To make sure the resulting tests run and compile, they are validated. These tests are iteratively improved by ASTER, which fixes any mistakes with focused, timely improvement.
- Coverage augmentation: Automated Test Case Generation(ASTER) finds untested code pathways and gives the LLMs instructions to create extra tests that target these locations in order to increase test efficacy and enhance overall code coverage.
Empirical validation and developer feedback
It used a number of models to assess ASTER, including Granite, the flagship model from IBM. Having conducted tests on GPT-4 turbo, Granite-8B, Llama3-8B, Granite-34B, CodeLlama-34B, and Llama3-70B. Also started with projects from the Defects4J dataset, which is made up of Java SE apps, however using a wide range of datasets. Additionally, it incorporated internal and open-source Java EE applications. During the evaluation, one can discovered:
Benefits of an LLM-based strategy driven by static analysis
In terms of coverage attained for Java SE projects, LLM-based test generation guided by static analysis is highly competitive with EvoSuite (the best conventional solution in terms of producing high-coverage tests for Java), being somewhat lower in certain situations (-7%) and significantly greater in other circumstances (4x-5x).
ASTER performs noticeably better than EvoSuite for Java EE projects (on average, surpassing it by 26.4%, 10.6%, and 18.5% in terms of line, branch, and method coverage attained) and is able to produce test cases for applications where current methods are unable to.
In comparison to CodaMosa, ASTER produces Python tests with greater coverage (+9.8%, +26.5%, and +22.5%) across all models.
The performance of smaller models is on par with that of larger variants
In contrast to larger models (here, Llama-70b and GPT-4), smaller models (in this instance, Granite-34b and Llama-3-8b) show competitive performance, with only 0.1%, 6.3%, and 2.7% loss in line, branch, and method coverage, respectively. Developers prefer models hosted internally or on local workstations, and the main advantages are price and addressing privacy concerns in enterprise environments that require on-premises solutions.
Everyone carried out an anonymous online survey at IBM to find out how developers felt about the usability and comprehension of tests created by ASTER as opposed to tests created by EvoSuite (or CodaMosa) or developers. A number of focal methods and two test cases for each technique are presented in the survey after a series of background questions. Each focal method and its test pair is then followed by a set of questions.
161 people who worked as software developers, QA engineers, lead solution architects, and research scientists responded to the poll. More than 70% of developers are prepared to add ASTER-generated tests with little to no modification to their test buckets, indicating that they prefer them over EvoSuite and CodaMosa tests in many ways.
Recognition and what’s next
At the 2025 International Conference on Software Engineering (ICSE), a leading platform for software-engineering research, the article outlining ASTER was accepted into the Software Engineering in Practice track and won the conference’s Distinguished article Award. Future research avenues include developing refined testing models to lower the cost of LLM interactions, expanding Automated Test Case Generation(ASTER) to other programming languages and testing levels, and investigating methods to enhance the generated tests’ fault-detection capabilities.