IBM used Benchpress, a new open-source quantum benchmarking software, to evaluate Qiskit’s performance against the top quantum SDKs in the world.
Based on IBM’s benchmarking utilising more than 1,000 tests created by top universities, national laboratories, and IBM, They concluded that Qiskit is the best quantum software development kit for creating and transpiling quantum circuits.
The outcomes of IBM’s comprehensive testing, which compared Qiskit to other top quantum software development kits, are being made public. A quantum SDK must be feature-complete enough to create and modify circuits while transpiling circuits with the optimal balance of speed and resultant circuit quality, and these tests benchmark some of the most crucial tasks for a quantum SDK. In these experiments, Qiskit was contrasted with Braket, Cirq, Staq, TKET, and BQSKit.
The outcomes? More tests might be completed using the Qiskit SDK than by any other SDK. On average, it used fewer two-qubit gates, was quicker, or both. It was the best-performing quantum SDK IBM evaluated, transpiling circuits 13 times quicker and using 24% fewer two-qubit gates than TKET, the second-best SDK.
IBM is making the suite of tests available as part of an open-source program called Benchpress, along with a report that summarises the findings. In order to extract valuable insights regarding different quantum SDKs, IBM developed Benchpress to assist industry stakeholders in reviewing and extending this study. IBM hope to provide an open platform for performance comparisons by making Benchpress open source, enabling users to conduct their own evaluations.
With Benchpress, customers will be able to keep an eye on the ongoing software development progress of quantum SDKs and make well-informed choices on which SDK to utilise for quantum computing. IBM invite you to check out and contribute to Benchpress yourself, and they will keep adding tests to the package.
Bringing the playing field level When IBM launched Qiskit SDK v1.0, the first major version of the Qiskit software development kit, earlier this year, Qiskit’s speed enhancements got underway. Although preliminary findings indicated that Qiskit v1.0 was a high-performance quantum SDK, IBM weren’t ready to declare that Qiskit was the best-performing quantum SDK globally until we conducted a formal comparison. In order to assess a quantum SDK’s capacity to perform its most crucial functions manipulating, constructing, synthesising, and transpiling circuits they put together a series of exacting tests.
The ability of a quantum SDK to construct and apply transformations to quantum circuits is measured via circuit creation and manipulation. While some quantum SDKs need manual circuit coding and circuit transformations, others include tools that automatically construct and modify circuits. In order to prevent the code from acting as a bottleneck to getting results rapidly, a performant quantum SDK should be able to construct and edit circuits quickly. Twelve of the exams consist of tasks involving the construction and manipulation of circuits; passing these examinations requires that the circuits be built and manipulated effectively in less than an hour.
The capacity of a quantum SDK to take a circuit as input and convert it into a form that can operate on real quantum hardware, such the IBM Quantum Heron processor, as well as abstract quantum hardware with arbitrary qubit layouts is known as synthesis and translation. Achieving success on these tests entails being able to transpile the circuits in less than an hour while using the fewest two-qubit gates possible.
About 90% of these tests are standardised benchmarking tests that are utilised by other members of the quantum community and are derived from open-source libraries. Benchpress, for instance, contrasts the capabilities of SDKs in producing QASMBench circuits created by Pacific Northwest National Labs, Feynman circuits created by Matthew Amy at the University of Waterloo, and Hamiltonian circuits created by a group of international physicists. IBM already benchmark Qiskit internally using the remaining tests.
Taking SDKs to the gym
According to their capabilities, IBM conducted testing on Braket, BQSKit, Cirq, Qiskit, and TKET for circuit creation and manipulation, and tests on BQSKit, Qiskit, Qiskit Transpiler Service, Staq, and TKET for transpilation. IBM choose not to assess software development kits like CUDA-Q whose feature sets don’t offer enough capabilities to conduct the tests.
Using two important metrics speed and quality IBM conducted a general comparison of SDKs’ performance in these tests. They timed the creation and manipulation of the circuit using the SDK, as well as the synthesis and transpile times of the circuit independently. Based on the depth and quantity of two-qubit gates, IBM also evaluated the quality of the generated circuits as well as the transpiled circuits. IBM noted if the SDK passed, failed, lacked the functionality to finish the test, or was predicted to fail the test for every test.
Qiskit passed every test that was feasible for it to pass, and it passed more tests than any other quantum SDK that was tested. Compared to TKET, the second-best SDK, Qiskit was 13 times quicker on average at transpiling and generated circuits with 24% fewer two-qubit gates. The other SDKs either had flaws that prohibited them from passing the test or were unable to finish it in the allotted hour.
New features for a performant Qiskit
What makes performance so crucial? Well, without a quantum software development kit, a quantum computer cannot do calculations. Researchers may be discouraged from investigating issues on genuine quantum systems if it takes too long to construct, modify, and transpile circuits. This may be the difference between running circuits immediately and waiting days to run circuits.
Using the finest tools available from each SDK, IBM conducted these tests honestly and as best IBM could. However, they acknowledge that they lack expertise in other quantum SDKs. In order to let anybody who is interested to examine and conduct these tests independently, IBM has made Benchpress open source. Beyond this initial release, IBM intend to expand Benchpress’s functionality through open-source initiatives to further establish the package’s standing as a reliable benchmarking suite for quantum SDKs.
New features for an improved Qiskit IBM keep adding features and enhancements to Qiskit to improve its performance. They refactored a large portion of the code into the Rust programming language, which is responsible for many of the efficiency improvements. Features for memory management and security are included into Rust to assist create software that is more effective and of higher quality. Qiskit devs really adore it.
Additionally, IBM’s group will shortly release a paper describing an algorithm called LightSABRE, which is an improvement on the SABRE algorithm used by Qiskit to transpile circuits that is, to map and route circuits onto physical devices while adhering to the connectivity constraints of the hardware and aiming for a minimum number of gates and circuit layers. When it first appeared in 2018, the original SABRE algorithm was regarded as a high-quality solution for this problem; but, in the age of bigger utility-scale quantum computers, its runtime has become a bottleneck. Compared to SABRE, LightSABRE is a more efficient scaling update developed in Rust that results in an 18.9% reduction in the cost of SWAP gates for the identical circuits.
Lastly, only the Qiskit Transpiler Service (QTS) consistently produced shorter, higher-quality circuits than the conventional Qiskit transpiler in IBM’s benchmarking experiments. IBM were able to accomplish notable increases in circuit quality, especially for bigger circuits, because of its AI-powered transpiler passes. When transferring to the heavy hex topology of IBM Quantum hardware, Artificial Intelligence transpiler passes resulted in an additional 24% decrease on average in 2-qubit gates and a 36% reduction on average in circuit depth for utility-scale circuits (i.e., 100+ qubits) in comparison to regular Qiskit SDK transpilation.
Building a performant software stack
Benchpress evaluates a quantum SDK’s circuit-building and circuit-transpile capabilities, but it also needs to execute circuits on actual hardware, which may represent various quantum computing architectures. IBM redesigned Qiskit at THINK 2024 to encompass IBM’s full-stack quantum software, which includes the Qiskit SDK, Qiskit Functions, Qiskit Transpiler Service, Qiskit Serverless, Qiskit Runtime Primitives, and the generative AI code helper tool Qiskit Code helper. When combined, these tools guarantee a software stack with great performance.
IBM monitors many indicators to assess Qiskit’s performance throughout the workflow execution phase, which gauges how effectively the program can operate circuits on quantum hardware. CLOPS, which counts the number of circuit layer operations per second that Qiskit Runtime + hardware can execute, is one of these metrics. The average error for every two-qubit gate in a layered circuit is also included, along with the error per stacked gate.
IBM is investigating how to use mirror circuits and other metrics to assess Primitive quality. Although IBM hasn’t formally defined these additional measures yet, the qiskit-device-benchmarking repository currently has tools that let users investigate these benchmarks. By releasing these measures, IBM encourage other quantum computer providers to provide comparable statistics and highlight the significance of hardware and software collaborating to provide a high-performance experience.
