“Research using supercomputers contributes to a modern and sus-tainable society,” explained Professor Holger Hanselka, President of KIT. “With the help of supercomputers, research in key areas, such as energy, environment, mobility, and medicine, will find new solu-tions faster. HoreKa thus fits perfectly into KIT's strategy to make significant contributions to managing the challenges facing society.”
“High-performance computing stands for rapid developments. With their ever increasing peak performances, supercomputers are crucial to both leading-edge research and the development of innovative products and processes in key economic areas. Thanks to institutions like KIT, Baden-Württemberg is a European leader in supercomputing and internationally competitive in this area. It is not only the impressive computing power of the machines, but also the concentrated methodological expertise that enables our computer-assisted top-level research to achieve breathtaking results,” said Baden-Wuerttemberg Science Minister Theresia Bauer.
The new “Hochleistungrechner Karlsruhe” (German for Karlsruhe high-performance computer), HoreKa for short, is expected to be one of the ten most powerful computers in Europe in 2021 and will have a computing power of more than 17 PetaFLOPS - 17 quadrillion computing operations per second, which corresponds to the performance of more than 150,000 laptops.
The system will be available to scientists from all over Germany. Thanks to the new supercomputer, researchers in the areas of materials sciences, earth system science, energy and mobility research in engineering, life sciences and particle and astroparticle physics will be able to gain a more detailed understanding of highly complex natural and technical processes. Of course, HoreKa can also be used by scientists studying the SARS-CoV-2 virus and, thus, will contribute to fighting the COVID-19 disease.
Computing and Storage Go Hand in Hand
With HoreKa, researchers can analyze far more details in larger systems, thereby extending normal simulations to so-called multiscale simulations. “Climate simulations and Earth system models, for example, will achieve much finer resolutions and, thus, a higher level of detail,” explains Professor Martin Frank, Director of the Scientific Computing Center (SCC) of KIT. “However, in addition to pure computing power, the demands on file systems are also increasing in terms of both capacity and latency. With HoreKa, we are consistently continuing the strategic orientation of SCC towards data-intensive computing”.
“Currently, highly diverse technical developments are taking place on the hardware market,” says Dr. Jennifer Schröter, Head of the High-performance Computing Group of SCC. “Our technical requirements were demanding, but the tendering process was deliberately kept open with respect to the exact technologies used to give our bidders the opportunity to design the most powerful systems possible.”
Two Innovative Chip Technologies – One High-Performance System
The result is an innovative hybrid system with almost 60.000 next-generation Intel Xeon Scalable Processor cores and 220 terabytes of main memory as well as 740 NVIDIA A100 Tensor Core GPUs. A non-blocking NVIDIA Mellanox InfiniBand HDR network with 200 GBit/s per port is used for communication between the nodes. Two Spectrum Scale parallel file systems offer a total storage capacity of more than 15 petabytes. The computer systems are made by Lenovo, while general contractor pro-com Datensysteme GmbH from Eislingen near Stuttgart is responsible for project coordination, system integration, delivery, and customer support.
“We are looking forward to putting this system into operation to-gether with our partners Lenovo and KIT and to handing it over to the users,” says Oliver Kill, Managing Director of pro-com. With HoreKa, pro-com is not only celebrating its 30th anniversary in 2020, but also the largest order in the company's history.
Machine Learning Supports Human Researchers
"Artificial intelligence and machine learning can dramatically accelerate scientific computations in the most significant areas of research, where the world’s problems are being solved,” says Marc Hamilton, Vice-President of Solutions Architecture and Engineering at NVIDIA. “NVIDIA A100 Tensor Core GPUs further support this accelerated research, and together with NVIDIA Mellanox InfiniBand technology, KIT’s new supercomputer will speed up scientific discovery for a broad range of important research."
Another central aspect in the system design has been the enormous amount of data generated by scientific research projects. Depending on the application, several hundred terabytes of data can be generated by a single simulation. To keep up with the growing amounts of data, the computing nodes, the InfiniBand network, and the parallel file systems of HoreKa each will provide up to four times the throughput of its predecessor system, ForHLR.
A multi-level data storage concept will guarantee high-throughput processing of data on external storage systems. With a data rate of up to 45 GByte/s, HoreKa will also be connected to the “Large Scale Data Facility” (LSDF) of the SCC which has been providing a modern infrastructure for the storage, administration, archiving, and analysis of research data since 2010.
Award-winning Energy Efficiency
HoreKa will be installed in a state-of-the-art data center constructed for its predecessor ForHLR on KIT‘s Campus North in 2015. The award-winning, energy-efficient hot water cooling concept based on the Lenovo Neptune Direct Water Cooling (DWC) technology will also be used for the new system.
The SCC employees chose the name HoreKa in reference to “GridKa”, the “Grid Computing Centre Karlsruhe”. It is also located at SCC and has successfully provided data storage and analysis capacities for large-scale experiments all over the world, including the Large Hadron Collider (LHC) at CERN in Switzerland, for more than 15 years. One of GridKa's greatest successes is its participation in the discovery of the Higgs particle in July 2012. GridKa is the largest and most powerful data center of its kind.
More information about HoreKa:
www.scc.kit.edu/en/services/horeka.php
More information on COVID-19 research at KIT:
www.kit.edu/kit/corona-pandemie-forschung-und-hilfsaktivitaeten-am-kit.php (in German)
www.scc.kit.edu/en/aboutus/13531.php
With bwUniCluster 2.0, KIT operates a second supercomputer as a state service:
www.scc.kit.edu/en/services/bwUniCluster_2.0.php
KIT Press Release: www.kit.edu/kit/english/pi_2020_035_kit-procures-new-supercomputer.php