Dr Emily Lane

Speeding up Basilisk with GPGPUs

A NeSI case study by Dr Emily Lane of NIWA.

Dr Emily Lane, a Coastal Modeller and Hydrodynamics Scientist at the National Institute of Water and Atmospheric Research (NIWA), describes her experience as a NeSI user.

“I first put in my NeSI application to call someone’s bluff. I knew broadly what NeSI was, but had not really looked into it much. My colleagues and I use numerical models to study tsunami and flood inundation and we had heard about codes that had been sped up orders of magnitude by moving the calculations onto GPGPUs (general-purpose graphics processing units). I liked the idea but no one at NIWA had the technical background to translate Basilisk, the code we use for modelling tsunami and flood inundation, to make it run on GPGPUs. We were discussing this at a meeting and someone suggested that I applied for a NeSI project. It sounded appealing but then others told me NeSI didn’t ‘do that kind of thing’ and would offer support but go no further than that. This was just at the start of NeSI 2 though so I figured what did I have to lose? Worst case scenario they’d just say no and I wouldn’t have lost anything.”

“So, I went onto NeSI’s website and filled out a form for a proposal. Next thing I know I get an email saying that John Rugis has been assigned as my dedicated contact person – gulp – I suddenly realised that it was almost Christmas and I was about to be away for most of the next month. A hastily sent email and we arranged to get back in touch towards the end of January. So in the New Year I arranged a Skype meeting with John. My expectations weren’t high but I came out of that first meeting practically bouncing off the walls in excitement! I’d explained a little about the code that I wanted to run and how we were keen to convert it into a GPGPU code. John seemed equally excited about working with us and what we could achieve together. He had ideas about who else he could bring into the project and the whole conversation seemed suffused with a can-do attitude.”

“Over the next few meetings we worked through the technical details of getting access to the Pan computing cluster and porting the code to it. Helped by the NeSI support crew we profiled the code and identified some obvious bottlenecks we could remove to start the speed-up process. Pretty soon we had a simple test-case of a trans-oceanic tsunami making its way across the Pacific being simulated on the Pan cluster.”

“This was the point we’d reached at the conclusion of the preliminary proposal. With assistance from John we filled out an application for a Merit project. We were excited to get approval, but then learned that John was stepping back from the project as he was taking up other opportunities. We were a little worried that we were losing such a valuable member of the team. Our replacement support person was Wolfgang Hayek, who works 50/50 NIWA and NeSI. John transitioned us over to Wolfgang and pretty soon we realised that although he didn’t start with the experience that John had, he made up for it with enthusiasm in spades. Even better, we still had John’s know-how in the background supporting Wolfgang, so it was the best of both worlds. Wolfgang quickly got stuck into the thick of the code and we likewise had our horizons expanded learning the ins and outs of GPGPUs.”

“One of the defining features of the numerical solver Basilisk is its adaptive grid – by subdividing squares into four smaller squares in areas where resolution is needed, Basilisk is able to ensure that the grid is resolved enough to calculate accurately where necessary, but is not too fine in other places where nothing is happening. This makes it efficient, but also harder to code for GPGPUs. As a starting point Wolfgang decided to first look at the Cartesian version of Basilisk (i.e. where the equations are solved on a regular square grid). This was an obvious starting step as it more closely aligns with earlier GPGPU algorithms. Another feature of Basilisk is its flexibility. We challenged Wolfgang to design a version that kept a lot of the flexibility while running on GPGPUs.”

"This shows a transpacific tsunami that originated near the Peru/Chile border at 4, 8, 12 and 16 hours. The top shows the grid, the bottom the maximum water elevation. You can see how the grid is only resolved where the action is."

“So far we are a year into the project and we’ve learnt a lot. We’ve got a Cartesian version of Basilisk that runs on GPGPUs with a good speed-up while still keeping a lot of the flexibility of the original programme. We’ve also learnt that some of the speed-up claims might be true going from an older, less efficient code to a modern GPGPU code, but aren’t realistic for a new, well-written code. With all this under our belt we are excited about the next challenge – seeing whether we can convert the adaptive grid version to GPGPUs. We’re pleased that we’ve got the support of NeSI and Wolfgang behind us because we couldn’t do it without them.”

Emily keeping an eye out for tsunamis in Lyttelton harbour while indulging in other interests...

Next Case Study

NeSI’s supercomputer helps shed light on future climate

“NeSI is providing the supercomputing infrastructure on which we are producing climate simulations, both globally and using a regional climate model. You cannot operate an Earth System Model without a supercomputer."