Helping communities anticipate flood events
The ability to forecast flood events in response to intense rainfall is critical to saving lives across New Zealand and preserving infrastructure. The 1938 event, when 21 people drowned on February 19 as a five metre high wall of water unexpectedly hit a sleeping camp of workers who had been building the Wairoa-Gisborne railway track, should serve as a reminder of the potentially devastating effects of flash floods on human life and property.
TopNet is a hydrological code developed at NIWA (the National Institute of Water and Atmospheric Research), which uses concepts of runoff generation controlled by subsurface water storage and topography. It combines a water balance model (see Figure 1) within each sub-catchment to simulate water flow, soil moisture, lake levels, discharge in rivers and streams over time, taking into account precipitation, snow, evaporation and plant transpiration (see Clark et al., 2008 for details of the implementation of TopNet). As part of the EcoConnect forecasting framework, it is run four times daily to predict potential flooding in New Zealand. The model is also being applied to assess the impact of climate change on water availability and to estimate changes in the probability of future flood events as global temperatures increase.
NeSI was called in to help NIWA improve TopNet’s performance on High Performance Computing (HPC) machines. The aim of this collaborative consultation project was to enable scientists to obtain results faster, run larger simulations with wider catchment areas, and execute bigger ensemble runs for improved uncertainty quantification. Support from NeSI was provided by Chris Scott and Alexander Pletzer through NeSI’s Scientific Programming Consultancy Service, while Gabriella Turek, Céline Cattoën and Christian Zammit provided scientific expertise from NIWA.
Our first task was to introduce a CMake build system, which supports out of source builds, automates to a large extent the search of library dependencies, and makes it easy to switch between compilers and compiler options. This revealed that performance improvements of 20% can sometimes be obtained simply by switching to a different compiler. In addition to improving the build system, NeSI also added a test harness based on CTest to ensure that code changes don’t affect well known results.
Applying profiling tools such as VTune allows you to identify bottlenecks in the code. For forecast type runs, we found that most of the execution time (66%) was spent writing large restart files and that a 3-4 times speedup (see Figure 2) could be achieved by writing the data in a single, large chunk instead of a large number of small chunks.
TopNet is sometimes used for parameter calibration, which in contrast to forecast simulations requires a much higher number of time steps but produces very little file output. Profiling these simulations showed that 68% of run time was spent in a specific subroutine (Figure 3), another hot spot in the code. Within this subroutine we identified sections that scaled proportionally to the number of time steps (the more time steps the slower) and were able to convert these to run in constant time, resulting in a 3-4 times speedup.
The top number in Figure 3 is the percentage of total run time spent in that function and its children, while the number in brackets is the time spent just in that function.
Much of the data written by TopNet is sparse and thus can easily take advantage of netCDF/HDF5’s on-the-fly compression – the amount of disk space used by sparse data can be greatly reduced at very little computational cost. By enabling compression in forecast simulations we achieved an approximately 80% reduction in file sizes. This will reduce load on the file system when many instances of TopNet run concurrently.
As a result of these changes TopNet has become easier to build and has inherited a more robust software development process. The ability to provide forecasts faster will enable scientists to run a larger number of TopNet ensemble runs and this will reduce uncertainty bounds. It will also allow researchers to extend the computational domain to cover all of New Zealand with a large number of river reaches. Faster calibration simulations will improve the choice of parameters in the empirical parts of the model. For both forecast and calibration runs, decreased run time execution will improve our ability to anticipate and act on future catastrophic flood events in New Zealand (see Fig. 4).
If you feel you could benefit from NeSI’s Scientific Programming Consultancy Service, or would like more information, please contact us at firstname.lastname@example.org.
Bandaragoda, C.; Tarboton, D.G.; Woods, R. 2004: Application of TOPNET in the distributed model intercomparison project. Journal of Hydrology, 298(1-4): 178-201. DOI:10.1016/j.jhydrol.2004.03.038
Clark, M. et al: Hydrological data assimilation with the ensemble Kalman filter: Use of streamflow observations to update states in a distributed hydrological model. Advances in Water Resources, 31: 1309-1324. DOI:10.1016/j.advwatres.2008.06.005