Simulation fails at forecast 007

1

Hi, I have a suite (u-dv923, just committed) that consistently fails at forecast 007. Here is the error message:

???
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 40
? Error from routine: NC_PUT_VAR_REAL_1D
? Error message: NetCDF: Index exceeds dimension bound : NF90_PUT_VAR
? Error from processor: 0
? Error number: 35
???

So it sounds like the issue is in writing one of the variables I request to netcdf - but I’m wondering how to tell which variable is causing the problem? For this suite, I’m creating output at specific timesteps in addition to the standard hourly output. To do that, I cloned a set of STASH requests and set their timestream to be the specific times of interest, and have them write to a new netcdf. The suite runs when I don’t request variables at the additional timesteps, and it also runs when I just requested one variable (theta after timestep) at the specific timesteps.

2

Hi, just following up on this, as I haven’t been able to resolve this issue yet.

3

Hi there.

I’m not sure how much I can help here since this is an NWP workflow and I primarily work with climate models.

Here are my thoughts:

The error is reproducible; that is, the crash happens at the same time step in each of the three runs of your ASE_km1p5_RAL3_um_fcst_007 task (20250513T1200Z, 20250514T1200Z, 20250515T1200Z), which is 10:23:00 on the 2nd day of the forecast. In my mind this points to some kind of memory error, i.e. that the files are ‘filling up’. This tallies with your comments that…

The suite runs when I don’t request variables at the additional timesteps, and it also runs when I just requested one variable (theta after timestep) at the specific timesteps.

My advice would be to try downgrading the precision of the NetCDf files…

image
image520×401 25.3 KB

… and/or reinitialise the files every, say, 24 hours rather than the 36 which it is currently set to (reinit_step)…

image
image749×518 32.8 KB

All the best.

Jonny