We present a collection of techniques for exploiting latent I/O asynchrony which can substantially improve performance in data-intensive parallel applications. Latent asynchrony refers to an application’s tolerance for decoupling ancillary operations from its core computation, and is a property of HPC codes not fully explored by current HPC I/O systems. Decoupling operations such as buffering and staging, reorganization, and format conversion in space and in time from core codes can shorten I/O phases, preserving valuable MPP compute cycles. We describe in this paper DataTaps, IOgraphs, and Metabots, three tools which allow HPC developers to implement decoupled I/O operations. Using these tools, asynchrony can be exploited by data generators which overlap computation with communication, and by data consumers that perform data conversion and reorganization out-of-band and on-demand. In the context of a data-intensive fusion simulation, we show that exploiting latent asynchrony through decoupling of operations can provide significant performance benefits.
Get full access to this article
View all access options for this article.
References
1.
Abbasi, H., Wolf, M., Schwan, K., Eisenhauer, G. and Hilton, A. ( 2004). Xchange: coupling parallel applications in a dynamic environment. In CLUSTER’04: Proceedings of the 2004 IEEE International Conference on Cluster Computing, Washington, DC, USA: IEEE Computer Society, pp. 471-480.
2.
Abbasi, H., Wolf, M., Zheng, F., Eisenhauer, G., Klasky, S. and Schwan, K. ( 2009). Scalable data staging services for petascale applications . In Proceedings of ACM High Performance Distributed Computing 2009, Munich, Germany: ACM, June.
3.
Beynon, M., Ferreira, R., Kurc, T.M., Sussman, A. and Saltz, J.H. ( 2000). Datacutter: Middleware for filtering very large scientific datasets on archival storage systems. In IEEE Symposium on Mass Storage Systems, pp. 119-134. [Online]. Available: citeseer.ist.psu.edu/beynon00datacutter.html .
4.
Brightwell, R., Hudson, T., Riesen, R. and Maccabe, A.B. ( 1999). The Portals 3.0 message passing interface. Technical report SAND99-2959. Sandia National Laboratories.
5.
Bustamante, F.E., Eisenhauer, G., Schwan, K. and Widener, P. ( 2000). Efficient wire formats for high performance computing . In Proceedings of Supercomputing 2000 (SC 2000) , Dallas, Texas, November.
6.
Cai, Z., Kumar, V. and Schwan, K. ( 2006). Self-regulating data streams for predictable high performance across dynamic network overlays. In Proceedings of the 15th IEEE International Symposium on High Performance Distributed Computing (HPDC 2006), Paris, France, June.
7.
CCA Forum. (2004). MxN Parallel Data Redistribution @ ORNL. January. [Online]. Available: http://www.csm.ornl.gov/cca/mxn
8.
Clayton, J. and McDowell, D. ( 2003). A multiscale multiplicative decomposition for elastoplasticity of polycrystals. International Journal of Plasticity19(9): 1401-1444.
9.
Cluster File Systems Inc. (2002). Lustre: a scalable, high-performance file system. November. [Online]. Available: http://www.lustre.org/docs/whitepaper.pdf
10.
Crawford, D., Ma, K.-L., Huang, M.-Y., Klasky, S. and Ethier, S. ( 2004). Visualizing gyrokinetic simulations. In IEEE Visualization, pp. 59-66.
11.
Deelman, E., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Blackburn, K., Lazzarini, A., Arbree, A., Cavanaugh, R. and Koranda, S. ( 2003). Mapping abstract complex workflows onto grid environments . J. Grid Comput. 1(1): 25-39.
Eisenhauer, G. ( 2010) The EVPath Library. [Online]. Available: http://www.cc.gatech.edu/systems/projects/EVPath
14.
Eisenhauer, G., Bustamante, F.E. and Schwan, K. ( 2002). Native data representation: An efficient wire format for high performance computing. IEEE Transactions on Parallel and Distributed Systems13(12).
15.
Eisenhauer, G., Bustamante, F. and Schwan, K. ( 2006). Publish-subscribe for high-performance computing. IEEE Internet Computing10(1): 40-47, Special issue on Message-Oriented Middleware.
16.
Eisenhauer, G. and Daley, L.K. ( 2000). Fast heterogenous binary data interchange. In Proceedings of the Heterogeneous Computing Workshop (HCW2000) , May 3-5, http://www.cc.gatech.edu/systems/papers/Eisenhauer00FHB.pdf
17.
Eisenhauer, G., Bustamente, F. and Schwan, K. ( 2000). Event services for high performance computing. In Proceedings of High Performance Distributed Computing (HPDC-2000).
Gibson, G.A., Nagle, D.F., Amiri, K., Butler, J., Chang, F.W., Gobioff, H., Hardin, C., Riedel, E., Rochberg, D. and Zelenka, J. ( 1998). A cost-effective, high-bandwidth storage architecture . SIGOPS Oper. Syst. Rev. 32(5): 92-103.
20.
Interagency Working Group on High End Computing. In HEC-IWG File Systems and I/O R&D Workshop. http://www.nitrd.gov/subcommittee/hec/workshop/20050816_storage
21.
The ITER Organization.The ITER Project. http://www.iter.org
22.
Jiao, X. and Heath, M.T. ( 2004). Common-refinement-based data transfer between nonmatching meshes in multiphysics simulations. International Journal for Numerical Methods in Engineering61(14): 2402-2427.
23.
Kumar, V., Cooper, B.F., Cai, Z., Eisenhauer, G. and Schwan, K. ( 2005). Resource-aware distributed stream management using dynamic overlays. In Proceedings of the 25th IEEE International Conference on Distributed Computing Systems (ICDCS-2005).
24.
Latham, R., Miller, N., Ross, R. and Carns, P. ( 2004). A next-generation parallel file system for Linux clusters . LinuxWorld2(1).
25.
Lofstead, J. and Schwan, K. ( 2005). Xchange: high performance data morphing in distributed applications. Report GIT-CC-05-14. Georgia Institute of Technology. [Online]. Available: ftp://ftp.cc.gatech.edu/pub/coc/tech_reports/2005/GIT-CC-05-14.pdf
26.
Lofstead, J., Schwan, K., Klasky, S., Podhorszki, N. and Jin, C. ( 2008). Flexible I/O and integration for scientific codes through the adaptable I/O system (ADIOS). In Proceedings of the 2008 IEEE Workshop of Challenges of Large Applications in Distributed Environments (CLADE), June.
27.
Ludscher, B., Altintas, I., Berkley, C., Higgins, D., Jaeger, E., Jones, M., Lee, E.A., Tao, J. and Zhao, Y. ( 2006). Scientific workflow management and the Kepler system: research articles. Concurr. Comput.: Pract. Exper. 18(10): 1039-1065.
28.
Oldfield, R.A., Womble, D.E. and Ober, C.C. ( 1998). Efficient parallel I/O in seismic imaging. The International Journal of High Performance Computing Applications12(3): 333-344. [Online]. Available: ftp://ftp.cs.dartmouth.edu/pub/raoldfi/salvo/salvoIO.ps.gz.
29.
Oldfield, R.A., Maccabe, A.B., Arunagiri, S., Kordenbrock, T., Riesen, R., Ward, L. and Widener, P. ( 2006). Lightweight I/O for scientific applications. In Proceedings of the 2006 IEEE Conference on Cluster Computing , Barcelona, Spain, September.
30.
Oliker, L., Carter, J., Wehner, M., Canning, A., Ethier, S., Mirin, A., Bala, G., Parks, D., Worley, P., Kitawaki, S. and Tsuda, Y. ( 2005). Leading computational methods on scalar and vector hec platforms. In Proceedings of IEEE/ACM Supercomputing2005.
31.
ORNL Visualization Task Group.Exploratory Visualization Environment for REsearch in Science and Technology (EVEREST) . http://www.csm.ornl.gov/viz
32.
Panasas Inc. (2003). Object-based storage architecture: defining a new generation of storage systems built on distributed, intelligent storage devices. October. [Online]. Available: http://www.panasas.com/docs/Object_Storage_Architecture_WP.pdf
33.
Pu, C., Schwan, K. and Walpole, J. ( 2001). Infosphere project: system support for information flow applications. SIGMOD Record30(1): 25-34.
34.
Rajasekar, A., Wan, M., Moore, R., Schroeder, W., Kremenek, G., Jagatheesan, A., Cowart, C., Zhu, B., Chen, S.-Y. and Olschanowsky, R. ( 2003). Storage Resource Broker-managing distributed data in a Grid. Computer Society of India Journal, Special Issue on SAN33(4): 42-54.
35.
Riedel, E., Faloutsos, C., Gibson, G.A. and Nagle, D. ( 2001). Active disks for large-scale data processing. IEEE Computer34(6): 68-74. [Online]. Available: http://www.computer.org/computer/co2001/r6068abs.htm
36.
Roth, P., Arnold, D. and Miller, B. ( 2003). Mrnet: a software-based multicast/reduction network for scalable tools. [Online]. Available: citeseer.ist.psu.edu/roth03mrnet.html .
37.
Schwan, K., Cooper, B.F., Eisenhauer, G., Gavrilovska, A., Wolf, M., Abbasi, H., Agarwala, S., Cai, Z., Kumar, V., Lofstead, J., Mansour, M., Seshasayee, B. and Widener, P. ( 2006). Auto-flow: autonomic information flows for critical information systems. In Autonomic Computing: Concepts, Infrastructure and Applications, edited by M. Parashar and S. Hariri.CRC Press.
38.
Thain, D., Tannenbaum, T. and Livny, M. ( 2005). Distributed computing in practice: the condor experience . Concurrency - Practice and Experience17(2-4): 323-356.
39.
Thakur, R., Gropp, W. and Lusk, E. ( 1999). Data sieving and collective I/O in ROMIO. In Symposium on the Frontiers of Massively Parallel Processing, vol. 0, p. 182.
40.
Weinberg, J., Jagatheesan, A., Ding, A., Faerman, M. and Hu, Y. ( 2004). Gridflow description, query, and execution at scec using the sdsc matrix. In HPDC’04: Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC’04), Washington, DC, USA: IEEE Computer Society, pp. 262-263.
41.
Weng, L., Agrawal, G., Catalyurek, U., Kur, T., Narayanan, S. and Saltz, J. ( 2004). An approach for automatic data virtualization. In Proceedings of the IEEE International Symposium on High Performance Distributed Computing (HPDC), pp. 24-33.
42.
Wolf, M., Cai, Z., Huang, W. and Schwan, K. ( 2002). Smartpointers: personalized scientific data portals in your hand. In Proceedings of IEEE/ACM Supercomputing 2002, November, http://www.sc-2002.org/paperspdfs/pap.pap304.pdf
43.
Wolf, M., Abbasi, H., Collins, B., Spain, D. and Schwan, K. ( 2005). Service augmentation for high end interactive data services . In Proceedings of the IEEE Cluster Computing Conference 2005 (Cluster’05), September.
44.
Yu, J. and Buyya, R. ( 2005). A taxonomy of scientific workflow systems for grid computing . SIGMOD Rec. 34(3): 44-49.
45.
Zhang, L., Parashar, M. and Klasky, S. ( 2006). Experiments with wide area data coupling using the Seine coupling framework. In LNCS: High Performance Computing-HiPC 2006, Springer Berlin, vol. 4297, pp. 229-241.
46.
Zang, T., Calinescu, R., Harris, S., Tsui, A., Kwiatkowska, M., Gibbons, J., Davies, J., Maccallum, P. and Caldas, C. ( 2008). Wsrf-based modeling of clinical trial information for collaborative cancer research. In Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2008), May.
