Contributions to high-performance big data computing

Geoffrey Fox, Judy Qiu, David Crandall, Gregor Von Laszewski, Oliver Beckstein, John Paden, Ioannis Paraskevakos, Shantenu Jha, Fusheng Wang, Madhav Marathe, Anil Vullikanti, Thomas Cheatham

Research output: Chapter in Book/Report/Conference proceedingChapter

4 Scopus citations


Our project is at the interface of Big Data and HPC - High- Performance Big Data computing and this paper describes a collaboration between 7 collaborating Universities at Arizona State, Indiana (lead), Kansas, Rutgers, Stony Brook, Virginia Tech, and Utah. It addresses the intersection of Highperformance and Big Data computing with several different application areas or communities driving the requirements for software systems and algorithms. We describe the base architecture, including the HPC-ABDS, High-Performance Computing enhanced Apache Big Data Stack, and an application use case study identifying key features that determine software and algorithm requirements. We summarize middleware including Harp-DAAL collective communication layer, Twister2 Big Data toolkit, and pilot jobs. Then we present the SPIDAL Scalable Parallel Interoperable Data Analytics Library and our work for it in core machinelearning, image processing and the application communities, Network science, Polar Science, Biomolecular Simulations, Pathology, and Spatial systems. We describe basic algorithms and their integration in end-to-end use cases.

Original languageEnglish (US)
Title of host publicationFuture Trends of HPC in a Disruptive Scenario
EditorsLucio Grandinetti, Gerhard R. Joubert, Kristel Michielsen, Seyedeh Leili Mirtaheri, Michela Taufer, Rio Yokota
PublisherIOS Press BV
Number of pages48
ISBN (Electronic)9781614999980
StatePublished - 2019

Publication series

NameAdvances in Parallel Computing
ISSN (Print)0927-5452
ISSN (Electronic)1879-808X


  • Big Data
  • Biomolecular simulations
  • Clouds
  • Graph Analytics
  • HPC
  • Network Science
  • Pathology
  • Polar Science

ASJC Scopus subject areas

  • Computer Science(all)


Dive into the research topics of 'Contributions to high-performance big data computing'. Together they form a unique fingerprint.

Cite this