Collectively clocking in at nearly 60 trillion particles, a newly released set of cosmological simulations is by far the biggest ever produced.
The simulation suite, dubbed AbacusSummit, will be instrumental in extracting secrets of the universe from upcoming surveys of the cosmos, its creators predict. They present AbacusSummit in several papers published October 25 in Monthly Notices of the Royal Astronomical Society.
AbacusSummit was produced by researchers at the Flatiron Institute’s Center for Computational Astrophysics (CCA) in New York City and the Center for Astrophysics | Harvard & Smithsonian. Made up of more than 160 simulations, it models how gravitational attraction causes particles in a box-shaped universe to move about. Such models, known as N-body simulations, capture the behavior of dark matter, which makes up most of the universe’s material and interacts only via gravity.
“This suite is so big that it probably has more particles than all the other N-body simulations that have ever been run combined — though that’s a hard statement to be certain of,” says Lehman Garrison, lead author of one of the new papers and a CCA research fellow.
Garrison led the development of the AbacusSummit simulations along with graduate student Nina Maksimova and astronomy professor Daniel Eisenstein, both of the Center for Astrophysics. The simulations ran on the U.S. Department of Energy’s Summit supercomputer at the Oak Ridge Leadership Computing Facility in Tennessee.
AbacusSummit will soon come in handy, as several surveys will produce maps of the cosmos with unprecedented detail in the coming years. These include the Dark Energy Spectroscopic Instrument, the Nancy Grace Roman Space Telescope and the Euclid spacecraft. One of the goals of these big-budget missions is to improve estimations of the cosmic and astrophysical parameters that determine how the universe behaves and how it looks.
Scientists will make those improved estimates by comparing the new observations to computer simulations of the universe with different values for the various parameters — such as the nature of the dark energy pulling the universe apart. With the improvements offered by the next-generation surveys comes the need for better simulations, Garrison says.
“The galaxy surveys are delivering tremendously detailed maps of the universe, and we need similarly ambitious simulations that cover a wide range of possible universes that we might live in,” he says. “AbacusSummit is the first suite of such simulations that has the breadth and fidelity to compare to these amazing observations.”
The project was daunting. N-body calculations — which attempt to compute the movements of objects, like planets, interacting gravitationally — have been among the foremost challenges in the field of physics since the days of Isaac Newton. They’re tricky because each object interacts with every other object, no matter how far apart they are. This means that as you add more objects, the number of interactions rapidly increases.
There is no general solution to the N-body problem for three or more massive bodies. The calculations available are simply approximations. A common approach is to freeze time, calculate the total force acting on each object, then nudge each one based on the net force it experiences. Time is then moved forward slightly, and the process repeats.
Using that approach, AbacusSummit handled colossal numbers of particles thanks to clever code, a new numerical method and lots of computing power. The Summit supercomputer was the world’s fastest at the time the team ran the calculations.
The team designed their codebase — called Abacus — to take full advantage of Summit’s parallel processing power, whereby multiple calculations can run simultaneously. Summit boasts lots of graphics processing units, or GPUs, that excel at parallel processing.
Running N-body calculations using parallel processing requires careful algorithm design because an entire simulation requires a substantial amount of memory to store. That means Abacus can’t just make copies of the simulation for different nodes of the supercomputer to work on. So the code instead divides each simulation into a grid. An initial calculation provides a fair approximation of the effects of distant particles at any given point in the simulation. (Distant particles play a much smaller role than nearby particles.) Abacus then groups nearby cells and splits them off so that the computer can work on each group independently, combining the approximation of distant particles with precise calculations of nearby particles.
For large simulations, the researchers found that Abacus’ approach offers a significant improvement on other N-body codebases, which divide the simulations irregularly based on the distribution of particles. The uniform divisions used by AbacusSummit make better use of parallel processing, the researchers report. Additionally, the regularity of Abacus’ grid approach allows a large amount of the distant-particle approximation to be computed before the simulation even starts.
Thanks to its design, Abacus can update 70 million particles per second per node of the Summit supercomputer (each particle represents a clump of dark matter with 3 billion times the mass of the sun). The code can even analyze a simulation as it’s running, looking for patches of dark matter indicative of the bright star-forming galaxies that are a focus of upcoming surveys.
“Our vision was to create this code to deliver the simulations that are needed for this particular new brand of galaxy survey,” says Garrison. “We wrote the code to do the simulations much faster and much more accurately than ever before.”
Eisenstein, who is a member of the Dark Energy Spectroscopic Instrument collaboration — which recently began its survey to map an unprecedented fraction of the universe — says he is eager to use Abacus in the future.
“Cosmology is leaping forward because of the multidisciplinary fusion of spectacular observations and state-of-the-art computing,” he says. “The coming decade promises to be a marvelous age in our study of the historical sweep of the universe.”
Additional co-creators of Abacus and AbacusSummit include Sihan Yuan of Stanford University, Philip Pinto of the University of Arizona, Sownak Bose of Durham University in England and Center for Astrophysics researchers Boryana Hadzhiyska, Thomas Satterthwaite and Douglas Ferrer. The simulations ran on the Summit supercomputer under an Advanced Scientific Computing Research Leadership Computing Challenge allocation.
