Parallel Test Execution Is Here: AWS CodeBuild Just Got Faster
Parallel Test Execution
That AWS CodeBuild now allows parallel test execution, which will allow you to run your test suites simultaneously and drastically cut down on build times.
The entire test time, including the time needed to provide the environments, was reduced from 35 minutes to 6 minutes with the demo project created for this post. The difference can be seen in these two AWS Management Console screenshots.
Sequential execution of the test suite
Parallel execution of the test suite
Running continuous integration (CI) at scale is significantly hampered by extremely long test times. The time needed to run thorough test suites can rise sharply with project complexity and team number, resulting in longer pipeline execution times. In addition to delaying the release of bug patches and new features, this reduces developer productivity by making them wait for build results before starting work.
Pipelines that take up to 60 minutes to run have failed at the last stage, necessitating a full restart and further delays. These protracted cycles have the potential to reduce developer confidence in the continuous integration process, increase dissatisfaction, and eventually slow down the software delivery cycle as a whole. Long-running tests can also result in resource contention, higher expenses due to wasted processing power, and decreased development process efficiency overall.
You can now run your tests concurrently across several build compute environments using CodeBuild’s parallel test execution feature. This feature uses a sharding technique in which a piece of your test suite is independently run by each build node. To decide which tests each node should perform, CodeBuild offers environment variables that specify the number of nodes in use as well as the overall number of nodes. Each node runs independently to complete its designated portion of your tests throughout build time; there is no control over the build node or collaboration between nodes.
Set the desired parallelism level and other pertinent values in the buildspec.xml’s batch fanout section to enable test splitting. Use the codebuild-tests-run utility in your build phase as well as the selected splitting method and the relevant test instructions.
The sharding approach you choose determines how the tests are divided. Codebuild-tests-run provides two methods for sharding:
- Equitable allocation: This method divides test files evenly among parallel test environments in chunks and arranges them alphabetically. Files may be reassigned across shards if the names or numbers of test files change.
- Stability: This method uses a consistent hashing algorithm to fix the distribution of tests across shards. When files are added or deleted, it preserves the current file-to-shard assignments.
When running tests concurrently, CodeBuild allows test reports to be automatically merged. CodeBuild streamlines result analysis by automatically combining test reports into a single test summary. The combined report eliminates the need for manual report processing by including aggregated pass/fail statuses, test durations, and failure information. To expedite test analysis, you can combine the merged findings with various reporting tools, see them in the CodeBuild console, or access them using the AWS Command Line Interface (AWS CLI).
Let’s look at how it works
Allow me to show you how to include parallel testing into a project. I made a very simple Python project with hundreds of tests for this demonstration. It used the command line to instruct Amazon Q Developer to generate a project and 1,800 test cases in order to expedite the process. Every test case takes one second to finish and is contained in a different file. It takes 30 minutes to run all the tests in order, not including environment provisioning time.
In this demonstration, at assess the test suite’s execution time while running it concurrently across ten compute environments.
The YAML file has three sections that need to be highlighted.
First, under batch, there is a build-fanout section. CodeBuild is instructed by the parallelism command how many test environments to execute concurrently. If any fanout construction job failure can be disregarded, it is indicated by the ignore-failure command.
Second, it execute tests using the codebuild-tests-run tool that comes pre-installed.
This command determines which tests need to be executed on the current node after receiving the entire list of test files.
- As a previously indicated, use the sharding-strategy argument to decide between a stable distribution and an equally dispersed distribution.
- To pass all of the files that are potential candidates for a run, use the files-search parameter. For performance reasons, we advise using the supplied codebuild-glob-search command; nevertheless, any file search tool, like find(1), will function.
- It use the test-command option to run the test command on the shard.
Finally, CodeBuild is instructed to gather and combine the test reports on every node by the reports section.
AWS then create a project and a batch build configuration for this project by opening the CodeBuild console. I won’t get into the specifics because there is nothing new to report. All the information you need to get started is in the manual. Batch builds are tested in parallel. Ensure that your project is set up to execute in batches.
Its now prepared to start the test suite’s execution. AWS may start the build in the console or add new code to my GitHub repository.
A few minutes later, it see a status report that details the various build phases, including the state of each test environment or shard.
AWS choose the Reports option to view the combined test reports after the test is finished.
All test data from all shards is combined in the Reports area, which also maintains the history for every build. To view the comprehensive report, I choose my most recent build from the Report history area.
AWS able to view both the aggregated and individual status for each of my 1,800 test cases, as anticipated. They are all passing in this demo, and the report is green.
The demo project’s 1,800 tests each take one second to finish. This test suite took thirty-five minutes to finish when I it one after the other. The test suite took six minutes to finish, including the time needed to provide the environments, when I ran it in parallel on 10 compute environments. 17.9% less time was spent on the parallel run than on the sequential one. The precise figures will change depending on your projects.
Additional things to know
All testing frameworks are compatible with this new feature. Examples for Django, Elixir, Go, Java (Maven), Javascript (Jest), Kotlin, PHPUnit, Pytest, Ruby (Cucumber), and Ruby (RSpec) are all included in the documentation.
The CODEBUILD_CURRENT_SHARD_FILES environment variable in the codebuild-tests-run CLI offers a versatile substitute for test frameworks that do not support space-separated lists. A list of test file paths for the current build shard is included in this variable, separated by newlines. It can be used to format test file names and adjust to various test framework needs.
By creating your own sharding script and utilizing the CODEBUILD_BATCH_BUILD_IDENTIFIER environment variable, which is set automatically in every build, you can further alter how tests are divided among environments. This method can be used to apply optimization or parallelization specific to a framework.
Pricing and availability
You can now finish your test suites in a fraction of the time that was previously needed with parallel test execution, which speeds up your development cycle and increases team productivity.
All three of CodeBuild’s compute modes on-demand, reserved capacity, and AWS Lambda compute allow for parallel test execution.
With no extra charge on top of the normal CodeBuild pricing for the compute resources utilized, this feature is currently accessible in all AWS Regions where CodeBuild is supported.
Encourage you to experiment with CodeBuild’s parallel test execution feature right now. For additional information and to begin parallelizing your tests, see the AWS CodeBuild documentation.
