A Massey University researcher has hit the big league in the world of high performance computing (HPC), and says her experience with NeSI played a key role in helping her get there.
Krista Steenbergen is a Postdoctoral Fellow at Massey University's Centre for Theoretical Chemistry and Physics (CTCP). In April 2017, the Partnership for Advanced Computing in Europe (PRACE) awarded Steenbergen 18.1 million computing hours on one of the world’s top supercomputers, the Marconi KNL system based at the CINECA supercomputing facility in Italy.
Her project collaborators include Massey colleagues Peter Schwerdtfeger and Elke Pahl, as well as Florent Calvo from the Université Grenoble Alpes in Grenoble, France. Their project was among the final 59 selected from 113 applications to PRACE’s 14th Call for Proposals for Project Access, and one of just two allocations awarded to projects outside of Europe.
Marconi is a Tier-0 system, co-designed by Cineca and Lenovo, with computational power of about 20 PFLOPS and a data storage capacity of more than 20 petabytes. To put that into perspective, using an example from Indiana University, in order to match what a 1 PFLOPS system can do in just one second, you'd have to perform one calculation every second for 31,688,765 years. In terms of storage, to hold just 1 petabyte of data you would need a stack of single-sided DVDs 290 metres tall, which is just 38 metres shy of the height of Auckland’s Sky Tower.
“In general, the class of HPC like Marconi is on a different level,” says Steenbergen. “It’s been built and configured assuming massively parallel applications. Even the way it’s run assumes that.”
This specialised environment is partly what inspired Steenbergen’s team to pursue their project.
“My two supervisors and I had been talking for a while about wanting to do a certain kind of simulation, and we knew that both our local cluster and the NeSI resources just wouldn’t be able to do that, so that’s where that PRACE allocation came from. I went out and I started looking for the types of resources that allow us to do that [simulation] at that scale, and that took me overseas.”
Armed with Marconi’s computing power, Steenbergen’s team is going to complete a set of simulations melting bulk mercury at a series of increasing pressures. They will compare the results from relativistic and non-relativistic models, and explore how relativistic effects alter or dictate mercury’s material properties under pressure.
Mercury’s thermodynamic properties have long been of interest to experimental and theoretical researchers as it is the only elemental metal that exists in the liquid state under standard conditions. Recent simulations have shown that mercury is a liquid at room temperature due to relativistic effects. To date, the interplay of relativistic effects, temperature and pressure have never been theoretically explored. The work of Steenbergen’s team over the next year aims to change that.
“It’s going to give us a lot of information that just isn’t known and really needs to be known,” Steenbergen says.
For example, most studies of mercury assume a 0 Kelvin (absolute zero) temperature to reduce the computational requirements and remove a number of potentially complicating effects, such as thermal effects. By studying mercury at temperature and systematically dialling up the pressure, Steenbergen’s team is expecting much more realistic results that could be ground-breaking and offer transferable insights into the properties of other important metals, like gold.
This leading edge approach was a big factor in why PRACE selected the project, but it wasn’t the only reason. Feedback from the selection committee specifically mentioned Steenbergen’s “excellent track record” with using HPC.
“That excellent track record in HPC experience is entirely the time that I have spent with NeSI,” she says.
Steenbergen’s NeSI use began in 2009 at Victoria University of Wellington, where she was pursuing her PhD. An IBM Blue Gene – one of the most powerful supercomputers available at the time – was available through NeSI and the University of Canterbury. Her PhD supervisor, Dr Nicola Gaston at the University of Auckland, planned research that incorporated high-level accurate quantum chemistry and physics codes to conduct specialised simulations. These computational demands went far beyond what could have been done on the smaller local clusters at the time. Without access to the Blue Gene through NeSI, Steenbergen’s project would have been impossible.
“We would have needed to take a completely different route and we would have had to change the project entirely,” she says. “I can credit NeSI with a big part of an incredibly successful PhD. We got a lot of great work done and it is something that my supervisor and I actually continue to collaborate on because it ended up to be such great research. I am incredibly grateful to NeSI.”
Now at Massey University, Steenbergen has projects running on NeSI’s Pan system as well as smaller local clusters. Not only have her NeSI allocations over the years been valuable, she says, but the NeSI support team itself has been just as important a resource.
“They’re the ones in the trenches who I write to, who support me, and who work with me,” she says. “I value them incredibly and I appreciate them. They have been nothing but supportive of all of my projects.”
During the span of her academic career, she has used supercomputer systems in the United States, Germany, New Zealand and now Italy. At each facility, and NeSI in particular, the technical staff have offered key support by helping compile code, improve job efficiency, and answer questions big or small.
“I know I can write to NeSI whenever I need to. They’re incredibly intelligent, incredibly supportive, and the turnaround time on solutions is just wonderful,” Steenbergen says. “From the outside perspective as a researcher, they work their tails off and they give exceptional support. I don’t think there are enough good things I can say about the NeSI team and how much I appreciate their support.”
Her Marconi allocation began in April and, as with all new computing environments, there has been a slight learning curve to getting started. But this is where her previous HPC experience has given her an edge.
“For every different supercomputer you use, there are different nuances and tweaks,” she says. “They’re all slightly different, but in general, having the HPC access, knowing what the basic hurdles will be, and having that experience put me hugely ahead of the curve. I was pretty much up and running in a couple of days.”
Steenbergen’s allocation on Marconi is valid for one year and comes with a prescribed usage schedule of 1.5 million core years every month. Since the system is built to support only massively parallel applications, she is also required to use full nodes, so her calculations needed to be able to scale up. This is different to how NeSI operates, where nodes are often shared and the systems are configured to accommodate a range of requirements around numbers of processors and amounts of memory or storage.
“For a system like Pan, where NeSI is trying to meet the needs of all its different users, that’s an incredible challenge, so that’s one of the reasons why I’m so appreciative of NeSI’s support,” Steenbergen says.
Looking ahead at her year on Marconi, Steenbergen is already predicting a slew of conference presentations and journal papers.
“It’s going to be very busy and very exciting. I’m trying to make it a priority that I give back in the way of research outputs at a scale comparable to what they’ve given me,” she says. “I think we’re going to end up with some amazing science out of this and I’m very excited about the things we’re going to learn.”
Will there be room in her busy schedule for a trip to CINECA in Italy?
“It’s not in the plans yet, but there’s no question I will do it in the future,” Steenbergen says. “It would be wonderful to meet Marconi.”
