Modelling Antarctic ice sheets

The below case study shares some of the technical details and outcomes of the scientific and HPC-focused programming support provided to a research project through NeSI’s Consultancy Service.

This service supports projects across a range of domains, with an aim to lift researchers’ productivity, efficiency, and skills in research computing. If you are interested to learn more or apply for Consultancy support, visit our Consultancy Service page.

Research background

The NZ SeaRise project is a $7.1 million, five-year research programme hosted at Victoria University of Wellington and led by Associate Professor Richard Levy and Professor Tim Naish. Funded by the Ministry of Business, Innovation and Employment, the overarching goal of the programme is to improve predictions of sea-level rise in Aotearoa New Zealand to 2100 and beyond.

Dr Stefan Jendersie is a Research Fellow who studies the interactions between sea ice, ice shelves, ice sheets and polar ocean conditions. He uses numerical modelling to simulate Antarctica’s ocean and ice under past, present and future environmental conditions. As part of the NZ SeaRise project, Stefan and his colleagues are aiming to couple the Parallel Ice Sheet Model (PISM) to the ice shelf version of the Regional Ocean Modeling System (ROMS). The sub-project's objective is to predict the influence of predicted global oceanic changes on the Antarctic ice sheet retreat on century scales.

Project challenges

• For some model configurations, input data needs to be read continuously at runtime. This is currently done by a single processor in a parallel run, which becomes a bottleneck/slows down code execution significantly at high model resolution, as the input data needs to be sent to all other processors in a parallel run, which can take a significant amount of time

• The researcher needs to couple the latest versions of the ROMS ocean model and CICE sea ice model and needed advice on how such coupling can be implemented

What was done

• Investigated if parallel IO, where data is read and written by more than a single processor, is beneficial

• Investigated other options to speed up model execution and provided advice on how to measure code performance, allowing to researcher to tune future configurations

• Provided advice on coupling models using the OASIS-MCT coupling library, including a detailed description of the process of setting up “toy models” (simple substitute models) to implement and test the coupling strategy - these toy models replace the actual ocean and sea ice code by simply loading precomputed ocean and sea ice data, allowing the research to focus on the details of the coupling process (e.g., which model fields need to be exchanged, in which order and at which simulation time, how the data can be interpolated between the different model grids, etc)

Main outcomes

• It turned out that parallel IO was not beneficial in the way it is currently implemented in the ocean model - resolving the IO bottleneck will likely require significant code changes such as using an IO server, which was beyond the scope of this consultancy

• Using lossless netCDF file compression for the large input data files reduced the storage footprint of this project substantially without affecting code performance, as less data needs to be read at runtime

• Achieved a speedup of 13% on 8% less cores on the Māui HPC compared to the original setup and with no unused cores on allocated nodes, and with lower memory consumption, enabling the researcher to use low-memory nodes on the HPC, which reduces queuing times and thus improves throughput

Researcher feedback

"With NeSI’s help, we were able to optimize our model to its maximum possible efficiency. While the overall efficiency gains were significant but not sufficient, the process of exploring the possible avenues (parallel I/O, tile masking, tile ordering) clarified the technological limitations of our model. Wolfgang Hayek put our model to the acid test, reviewed each and every component and provided a wealth of revealing performance statistics.

"Thanks to the NeSI Consultancy, I was able to learn and understand much more about massively parallelized computation and its underlying logic. We can now confidently say that our current model operates at its absolute technological limit. Even more so, the positive and very informative Consultancy sessions established a very solid knowledge base for targeted future upgrades of our modeling technology. We now know exactly what the shortcomings are and where to focus our development efforts most effectively."

- Stefan Jendersie, Research Fellow, Victoria University of Wellington

Do you have a research project that could benefit from working with NeSI research software engineers or a data engineer? Learn more about what kind of support they can offer and get in touch by emailing support@nesi.org.nz.