Due to the increasing diversity of high-performance computing architectures, researchers and practitioners are increasingly interested in comparing a code’s performance and scalability across different platforms. However, there is a lack of available guidance on how to actually set up and analyze such cross-platform studies. In this paper, we contend that the natural base unit of computing for such studies is a single compute node on each platform and offer guidance in setting up, running, and analyzing node-to-node scaling studies. We propose templates for presenting scaling results of these studies and provide several case studies highlighting the benefits of this approach.
AdhiantoLBanerjeeSFaganM, et al. (2010) HPCToolkit: tools for performance analysis of optimized parallel programs. Concurrency and Computation: Practice and Experience22(6): 685–701.
2.
AhnDHBassNChuA, et al. (2020) Flux: overcoming scheduling challenges for exascale workflows. Future Generation Computer Systems110: 202–213.
3.
AlpayASoproniBWu¨nscheH, et al. (2022) Exploring the possibility of a hipSYCL-based implementation of oneAPI. In: Proceedings of the 10th International Workshop on OpenCL, IWOCL ’22. New York, NY, USA: Association for Computing Machinery. ISBN 9781450396585.
4.
AmdahlGM (1967) Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the April 18-20, 1967, Spring Joint Computer Conference, AFIPS ’67 (Spring). New York, NY, USA: Association for Computing Machinery, 483–485.
5.
AndersonRBlackABusbyL, et al. (2020) The multiphysics on advanced platforms project (MAPP). Technical report.
6.
AndersonRAndrejJBarkerA, et al. (2021) MFEM: a modular finite element methods library. Computers & Mathematics with Applications81: 42–74.
BeckingsaleDABurmarkJHornungR, et al. (2019) RAJA: portable performance for large-scale scientific applications. In: 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). IEEE, 71–81.
9.
BeckingsaleDAMcFaddenMJDahmJPS, et al. (2020) Umpire: application-focused management and coordination of complex hierarchical memory. IBM Journal of Research and Development64(3/4): 1.
10.
BhateleABrinkSGamblinT (2019) Hatchet: pruning the overgrowth in parallel profiles. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’19. New York, NY, USA: Association for Computing Machinery.
11.
BoehmeDGamblinTBeckingsaleD, et al. (2016) Caliper: performance introspection for HPC software stacks. In: Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’16. IEEE Computer Society, 47:1–47:11. LLNL-CONF-699263.
12.
BoehmeDAschwandenPPearceO, et al. (2021) Ubiquitous performance analysis. In: ChamberlainBLVarbanescuALLtaiefH, et al. (eds) Proceedings ISC High Performance, ISC-HPC ’21. Springer International Publishing, 431–449.
13.
BrinkSMcKinseyMBoehmeD, et al. (2023) Thicket: seeing the performance experiment forest for the individual run trees. In: Proceedings of the 32nd International Symposium on High- Performance Parallel and Distributed Computing, HPDC ’23. New York, NY, USA: Association: Computing Machinery. ISBN 9798400701559, 281–293.
14.
BrowneSDongarraJGarnerN, et al. (2000) A portable programming interface for performance evaluation on modern processors. The International Journal of High Performance Computing Applications14(3): 189–204.
15.
CalotoiuAHoeflerTPokeM, et al. (2013) Using automated performance modeling to find scalability bugs in complex codes. In: Proceedings Of the ACM/IEEE Conference on Supercomputing (SC13). Denver, CO, USA: ACM, 1–12.
16.
CappsACarsonRCorbettB, et al. (2017–2024) Axom: CS infrastructure components for HPC applications. URL. https://github.com/llnl/axom
Di NataleFBhatiaHCarpenterTS, et al. (2019) A massively parallel infrastructure for adaptive multiscale simulations: modeling RAS initiation pathway for cancer. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’19. New York, NY, USA: Association for Computing Machinery.
19.
DingNWilliamsS (2019) An instruction roofline model for GPUs. In: 2019 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). IEEE, 7–18.
20.
DobrevVKnuppPKolevT, et al. (2019) The target-matrix optimization paradigm for high-order meshes. SIAM Journal on Scientific Computing41(1): B50–B68.
21.
EagerDZahorjanJLazowskaE (1989) Speedup versus efficiency in parallel systems. IEEE Transactions on Computers38(3): 408–423.
22.
EdwardsHCTrottCRSunderlandD (2014) Kokkos: enabling manycore performance portability through polymorphic memory access patterns. Journal of Parallel and Distributed Computing74(12): 3202–3216.
23.
GramaAYGuptaAKumarV (1993) Isoefficiency: measuring the scalability of parallel algorithms and architectures. IEEE Parallel & Distributed Technology: Systems & Applications1: 12–21.
24.
GustafsonJL (1988) Reevaluating Amdahl’s law. Communications of the ACM31(5): 532–533.
25.
GustafsonJ (1990) Fixed time, tiered memory, and superlinear speedup. In: Proceedings of the Fifth Distributed Memory Computing Conference, Vol. 2, 1255–1260.
26.
HertenA (2023) Many cores, many models: GPU programming model vs. Vendor compatibility overview. In: Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC- W ’23. New York, NY, USA: Association for Computing Machinery, 1019–1026.
27.
HoeflerTBelliR (2015) Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance resultsSupercomputing.
28.
HornungRDKeaslerJA (2014) The RAJA portability layer: overview and status.
29.
HorwitzJ (2024) Estimating the carbon footprint of computa-tional fluid dynamics. Physics of Fluids36(4): 045109.
JacobsenDBirdRF (2023) Ramble: a flexible, extensible, and composable experimentation framework. In: Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23. New York, NY, USA: Association for Computing Machinery, 600–608.
32.
KnuppP (2012) Introducing the target-matrix paradigm for mesh optimization via node-movement. Engineering with Computers28(4): 419–429.
PatkiTBertschAKarlinI, et al. (2021) Monitoring large scale supercomputers: a case study with the lassen supercomputer. In: 2021 IEEE International Conference on Cluster Computing (CLUSTER), 468–480.
40.
PearceOScottABeckerG, et al. (2023) Towards collaborative continuous benchmarking for HPC. In: Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23. New York, NY, USA: Association for Computing Machinery, 627–635.
41.
PetersonJLAtheyKBremerPT, et al. (2019) Merlin: enabling machine learning-ready HPC ensembles. In: Technical Report. Livermore, CA (USA): Lawrence Livermore National Lab.(LLNL).
42.
RichardsDFRyujinBSBartonN, et al. (2021) Enhancements Supporting IC Usage of PEM Libraries on Next-Gen Platforms. Livermore, CA (USA): Lawrence Livermore National Lab.(LLNL). Technical report.
43.
RyujinBSVargasAKarlinI, et al. (2022) Understanding power and energy utilization in large scale production physics simulation codes.
44.
ShendeSSMalonyAD (2006) The tau parallel performance system. The International Journal of High Performance Computing Applications20(2): 287–311.
45.
SinghJHennessyJGuptaA (1993) Scaling parallel programs for multiprocessors: methodology and examples. Computer26(7): 42–50.
46.
StittTBelcherKCamposA, et al. (2024) Performance portable graphics processing unit acceleration of a high- order finite element multiphysics application. Journal of Fluids Engineering146(4): 041102.
47.
SunXHGustafsonJ (1991) Toward a better parallel performance metric. Parallel Computing17(10): 1093–1109, Benchmarking of high performance supercomputers.
48.
SunXHNiL (1990) Another view on parallel speedup. In: Supercomputing ’90:Proceedings of the 1990 ACM/IEEE Conference on Supercomputing, 324–333.
TrottCRLebrun-Grandie´DArndtD, et al. (2022) Kokkos 3: programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems33(4): 805–817.
51.
VargasAStittTMWeissK, et al. (2022) Matrix-free approaches for GPU acceleration of a high-order finite element hydrodynamics application using MFEM, Umpire, and RAJA. The International Journal of High Performance Computing Applications36(4): 492–509.
52.
YasinA (2014) A top-down method for performance analysis and counters architecture. In: 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). CA, USA: IEEE, 35–44.
