Application-Oriented Performance Benchmarks for Quantum …

1 Application-Oriented Performance Benchmarks for Quantum Computing Thomas Lubinski,1, 2 Sonika Johri,3 Paul Varosy,4 Jeremiah Coleman,5 Luning Zhao,3 Jason Necaise,6 Charles H. Baldwin,7 Karl Mayer,7 and Timothy Proctor8. ( Quantum Economic Development Consortium (QED-C) collaboration) . 1. Quantum Circuits Inc, 25 Science Park, New Haven, CT 06511. 2. QED-C Technical Advisory Committee on Standards and Performance Benchmarks Chairman 3. IonQ Inc, 4505 Campus Dr, College Park, MD 20740, USA. 4. Department of Physics, Colorado School of Mines, Golden, CO 80401, USA. 5. Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ, 08544, USA. 6. D-Wave Systems, Burnaby, British Columbia, Canada, V5G 4M9, Canada 7. Quantinuum, 303 S. Technology Ct, Broomfield, CO 80021, USA.

2 8. Quantum Performance Laboratory, Sandia National Laboratories, [quant-ph] 30 Dec 2021. Albuquerque, NM 87185, USA and Livermore, CA 94550, USA. (Dated: January 3, 2022). In this work we introduce an open source suite of Quantum Application-Oriented Performance Benchmarks that is designed to measure the effectiveness of Quantum computing hardware at executing Quantum applications. These Benchmarks probe a Quantum computer's Performance on various algorithms and small applications as the problem size is varied, by mapping out the fidelity of the results as a function of circuit width and depth using the framework of volumetric benchmarking. In addition to estimating the fidelity of results generated by Quantum execution, the suite is designed to benchmark certain aspects of the execution pipeline in order to provide end-users with a practical measure of both the quality of and the time to solution.

3 Our methodology is constructed to anticipate advances in Quantum computing hardware that are likely to emerge in the next five years. This benchmarking suite is designed to be readily accessible to a broad audience of users and provides Benchmarks that correspond to many well-known Quantum computing algorithms. CONTENTS VI. Highlights and Impressions 15. I. Introduction 2 VII. Summary and Conclusions 18. II. Background 3 Code Availability 19. A. Component-Level Specifications 3. Acknowledgement 19. B. Quantum Volume 4. C. Volumetric Benchmarks 4. A. Selected Algorithms and Applications 20. 1. Shallow Simple Oracle Based Algorithms 20. III. Application-Oriented Benchmarks 5. 2. Quantum Fourier Transform 21. A. The Benchmarking Suite 5. 3. Grover's Search Algorithm 22. B. Volumetric Positioning 5.

4 4. Phase and Amplitude Estimation 23. C. Circuit Depth 6. 5. Hamiltonian Simulation 23. D. Circuit Fidelity 7. 6. Monte Carlo Sampling 24. IV. Benchmark Results and Analysis 7 7. Variational Quantum Eigensolver 25. A. Benchmark Results on Quantum Simulator 8 8. Shor's Order Finding 27. B. Benchmark Results on Quantum Hardware 8. B. Advances in Quantum Computers 28. C. Comparison to Generic Benchmarks 12. 1. Mid-Circuit Measurements 28. V. Measuring Execution Time 13 2. Circuit Parameterization 28. A. Total Execution Time 14 3. Multiply-Controlled Gates 29. B. Quantum Execution Time 14 4. Close Classical/ Quantum Integration 29. 5. Qubit Connectivity Improvements 30. References 31. This work was sponsored by the Quantum Economic Development Con- sortium (QED-C) and was performed under the auspices of the QED-C.

5 Technical Advisory Committee on Standards and Performance Benchmarks . The authors acknowledge many committee members for their input to and feedback on the project and this manuscript. 2. Volumetric Positioning - All Applications (Merged) Volumetric Positioning - All Applications (Merged) Volumetric Positioning - All Applications (Merged). Device=qasm_simulator 2021-09-14 18:26:04 UTC Device=qasm_simulator 2021-09-22 23:40:19 UTC Device=qasm_simulator 2021-09-22 23:41:00 UTC. 17 17 17. 16 i (1) 16 16 tion iran imula -Vaz (2) ian S. 15 stein 15 ilton Bern ch-Jozsa rans form 15 Ham e u ts if t er T 14 D. en S. h 14 Fouri n (1) 14 ). Hidd ntum tio rans form ing (2. Qua Estima T d 13 13 e Fouri e r 13 er Fin Phas s Ord ntum Shor' ). 12 Qua 12 (1. u la tion ). Sim (2. pling Avg Result Fidelity Avg Result Fidelity 11 11 11 VQE Sam on arlo Result Fidelity mati Circuit Width Circuit Width Circuit Width ti te C h 10 e Es 10 M on Searc litud er's Amp 9 G ro v 9 9.

6 8 8. 7 7 7. 6 6 i (2). 5 iran 5 5. -Vaz stein 4 Bern 4 4. 3 3 3. 2 2 2. QV=32 QV=32 QV=32. 1 1 1. 1K. 3K. 6K. K. K. 13 K. 7K. 66 K. 2K. 1. 2. 5. 11. 23. 52. 3. 9. 9. 1M. 3M. 7M. 1K. 2K. 3K. 5K. 8K. K. 1K. 2K. 3K. 6K. K. K. K. K. 1. 2. 3. 4. 7. 10. 17. 27. 43. 69. 0. 6. 1. 0. 1. 1. 2. 3. 5. 9. 15. 26. 44. 77. 2. 7. 0. 0. 11. 24. 54. 13. 28. 62. 1. 11. 17. 28. 45. 72. 12. 13. 22. 39. 67. 10. 17. 30. 51. 30. Circuit Depth Circuit Depth Circuit Depth FIG. 1. Quantum Application-Oriented Performance Benchmarks . The results of executing our Quantum Application-Oriented Performance benchmarking suite on a simulator of a noisy Quantum computer, with results split into Benchmarks based on three loose categories of algorithm: tutorial, subroutine, and functional. For each benchmark, circuits are run with a variety of widths, corresponding to the problem size, here ranging from 2 to 12 qubits.

7 The result fidelity, a measure of the result quality , is computed for each circuit execution, and is shown as a colored square positioned at the corresponding circuit's width and normalized depth. Results for circuits of equal width and similar depth are averaged together. The results of the Application-Oriented Benchmarks are shown on top of a volumetric background' (grey-scale squares). which extrapolates from a device's Quantum volume (here, 32) to predict the region in which a circuit's result fidelity will be above 1/2 (the grey squares). I. INTRODUCTION useful size of a specific Quantum computer and summarize it in one number: the Quantum volume. The Quantum volume has been widely adopted and reported by the community. How- Over the past decade, Quantum computers have evolved ever, due to the complexity of errors in Quantum hardware, from exploratory physics experiments into cloud-accessible neither a device's Quantum volume nor any other single metric hardware, heralding their transformation into early-stage com- is likely to accurately predict its Performance on all applica- mercial products.

8 A broad audience of users has now begun tions [7, 8]. There is thus a pressing need for a diverse set of to explore and interact with Quantum computers, inspiring re- Application-Oriented metrics and Benchmarks that test the per- search into practical Quantum algorithms [1] and garnering formance of Quantum computers on practically relevant tasks. attention from the academic, government, and business com- Such Benchmarks will enable hardware developers to quantify munities. With recent demonstrations of Quantum advantage their progress in commercially relevant ways, and will make [2, 3], it seems increasingly likely that Quantum computers it possible for end users to more accurately predict how the could soon outperform their classical counterparts for practi- available hardware will perform on their application.

9 Cally relevant tasks. Unlike contemporary classical computers, In this paper, we introduce an extensible suite of Quantum though, the capabilities of current Quantum computers are lim- Application-Oriented Performance Benchmarks , complementing ited primarily by errors, not by their size or speed. other recent methods [11 17] that use, , small chemistry Quantum computers can experience a wide range of complex problems or basic Quantum circuits as Benchmarks . Demon- errors. Many of these errors result from noise in or miscalibra- strated in Figure 1, each benchmark in our suite is derived from tions of the lowest-level components in the system Quantum an algorithm or application and specifies a scalable family of gate operations and the qubits they act on. Component-level Quantum circuits.

10 The Benchmarks are designed to evaluate Benchmarks and characterization protocols, such as random- the capability of Quantum hardware to successfully execute ized benchmarking [4, 5] or gate set tomography [6], can pro- a meaningful computational task, and are intended to reflect vide insight into the type and magnitude of these errors. These likely use cases for Quantum computers. The benchmarking tools are critical for experimental efforts to improve hardware. suite is available as a public, open-source repository with ex- But users seeking to run an application on a Quantum com- tensive documentation [18]. Validated implementations of our puter are typically concerned not with low-level details, but Benchmarks are supplied in multiple common Quantum pro- rather how likely their application is to execute successfully.