I am putting this user query on the helpdesk with some explanation of what is going on and how to fix it, as it might be useful for others.
A user has reported their coupled model has been repeatedly timing out and they cannot make any progress with it.
Archer2 has been slow recently which is likely to do with file system contention as users copy off data at the end of the service. So we increasing the model time limit to hopefully avoid future timeouts.
I suspect that the first time out has left the model in an incomplete state where it can’t recover from.
Looking at the timestamps of the files in the work/coupled directory we can see that the model starts up but then appears to hang for the rest of the run time.
For reference, the model cycle we are trying to run here is 2009-04-01 to 2009-07-30 (we have a 360d calendar). This is an ensemble suite so there are references to coupled_02 and History_Data_02.
The UM log file in work/20090401T0000Z/coupled_02/pe_output shows the UM component has completed succesfully but it is starting and ending at :
Atm_Step: Timestep 712800 Model time: 2009-07-01 00:00:00
The NEMO ocean component is OK, it is starting from the end of the previous cycle. In the ocean.output file we see this:
*** Info read in restart :
previous time-step : 313920
*** restart option
nrstdt = 2 : calendar parameters read in restart
*** Info used values :
date ndastp : 20090330
This is why the model hangs. The ocean is waiting to couple to the atmosphere but the atmosphere has already finished.
Why has the UM got the wrong start date? It picks this up from the $RUNID.xhist file in share/data/History_Data_02. If you look in that file you will see these variables:
H_STEPIM = 712800
CHECKPOINT_DUMP_IM = /work/n02/n02/vmascolo2/cylc-run/u-ec911/run2/share/data/History_Data_02/ec911-02a.da20090701_00
What we need to do is update this to start from the correct time point. We can get the correct history file from the previous cycle’s work directory:
work/20090101T0000Z/coupled_02/history_archive/
There will be one of more temp_hist.* files. Take the latest one and copy it to $RUNID.xhist in share/data/History_Data. If you want to be safe, back up the original .xhist first.
Check that the CHECKPOINT_DUMP_IM variable matches the start of the cycle you are running.
Then you should be able to re-trigger and it should now run.