Dear CMS team,
My suite u-ea645 has failed in postproc_atmos on cycle 185210 due to a corrupt file (as in Error at postproc_atmos - Unified Model - NCAS Modelling Support). I have tried retriggering this cycle but as the other postprocessing tasks have run, the ocean and cice restart files have been removed. I am therefore trying to restart from where I have archived dumps - 1852 01.
Is it possible to move these archived restarts back into the History_data directory and retrigger the run from the 1852 01 cycle or should I do a full restart (following tips_CRgeneral – MOCI) using cylc vip (I assume this would create a run2?)
Many thanks for your help,
Lauren
Hi Lauren.
Yes you can certainly bring back your restarts and then re-trigger using flow=none to isolate the task(s) that need re-running.
You’ll need to…
-
Edit the ainitial, NEMO_START, NEMO_ICEBERGS and CICE_START environment variables in ~/roses/u-ea645/app/{um,nemo_cice}/rose-app.conf.
-
‘Reinstall’ the workflow using cylc reinstall or cylc vr (validate, reinstall).
-
Trigger the task(s) that need rerunning using flow=none.
You can read about this here or check out the cylc trigger docs by cylc trigger -h (see below).
Also, yes, cylc vip will start a run2 from the start; VIP is short for validate, install, play. 
Cheers
Jonny
$cylc trigger -h | head
Usage: cylc trigger [OPTIONS] ID ...
Trigger a group of one or more tasks, respecting dependencies among them.
Prerequisites on tasks outside of the group will be satisfied automatically.
Tasks will be removed if necessary to allow re-run without intervention, so
triggered tasks that are preparing, submitted, or running may be killed.
Tasks that lead into a group will run immediately even if the workflow is
… also there is some information on restarting failing workflows here…
https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#RestartingFailingSuites
Thanks to @RosalynHatcher for the tip!
Jonny
… and note that the tips in the MOSRS webpage should negate the need to change the atmosphere, ocean, etc, restarts as I noted above; changes in them are really only necessitated when starting a new run. Triggering with flow=none is still relevant though.
Anyway, let us know how you get on!
Cheers
Jonny
Hi Jonny,
Thank you for the info. I’ve attempted to start from the previous January archived dumps but I’m currently getting new runs that then can’t find the executables. Could you point me to where I’ve gone wrong? Why do I need the flow=None? Or, alternatively, should I build the executables for this cycle?
I also have another run (u-ea644) that is failing the coupled task due to time limits. I’ve upped the wallclock by an hour but it’s still running out of time. Is there anything else going on?
Thanks,
Lauren
Hi Lauren,
Can you please change the permissions on your PUMA & ARCHER2 directories so that we can see into them?
chmod -R g+rX /home/n02/n02/<userid>
chmod -R g+rX /work/n02/n02/<userid>
What steps did you take to try and roll back to the January?
I’m guessing not by following the https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#RestartingFailingSuites instructions?
Did you cylc vip to start a new run?
Regarding the coupled task running out of time. We have had occurences of slow running on some of the compute nodes. I’ll take a look when I can see your files to confirm.
Cheers,
Ros
Hi Ros,
Thank you. I followed the instructions under ‘restarting from archived restarts’ up until point 10: run as NRUN. I have then tried both cylc vr for the original run1 and retriggered the 1852-01 cycle, and cylc vips - run 2 I hadn’t turned off the build and recon, run3 latest attempt.
Thanks a lot,
Lauren
Hi Lauren,
If you’ve got all the18521001 restart dumps I’m pretty sure we should be able to patch run1 up and then just retrigger the 18521001T0000Z/coupled task.
But if you’ve made changes to the suite now it’s probably going to be best to continue along the track you’ve now started. With Cylc 8 you either have to turn the build on or manually add in a symlink to it, since a new run uses a completely new run directory and knows nothing about any previous runs.
To go the symlink root, on ARCHER2
cd /work/n02/n02/eelrm/cylc-run/u-ea645/run3/share
ln -s ../../run1/share/fcm_make_um fcm_make_um
ln -s ../../run1/share/fcm_make_ocean fcm_make_ocean
Then retrigger the coupled task.
Cheers,
Ros
Hi Ros,
Thanks. I’ve given the symlink option a go but am now getting a Mismatch between model_basis_time read from namelist and validity time read. Where is the 1850 date coming from?
Thanks,
Lauren
HI Lauren.
I’m just doing a test of this setup. The date mismatch error will presumably be because of the lack of the recon step which, in the ‘normal’ case overwrites the dump’s date (1/1/3075) with the required workflow BASIS time.
Cheers
Jonny
Hi Lauren.
I’ve done some digging and certainly a cause of your issue that is causing your date-time mismatch is that your restart file /work/n02/n02/eelrm/Dumps/forcevol/ea645a.da18520101_00 appears to be erroneous.
From inspecting this file and comparing its output (I’ve looked at QCF and U wind) it is identical to /work/n02/n02/eelrm/Dumps/forcevol/by230a.da30750101_00 and has the 1/1/3075 date stamp.
What I think has happened is that the reconfiguration step was run (replacing the astart parameter in ~/roses/u-ea645/app/um/rose-app.conf with the ea645a.da18520101_00 path above. This will (quietly) overwrite a file with the same name in the same directory.
My advice now would be to either…
- Get back the archived restart for the atmosphere and try again.
- Run a fresh simulation from
18500101T0000Z since you are so close to the beginning of the run.
Hope that helps.
Jonny
Hi Jonny,
Thanks a lot - makes sense. I’ve reverted the suite and submitted a fresh simulation. Are there any other updates on the wallclock issue for u-ea644?
Many thanks,
Lauren
Hi Lauren.
The wallclock issue with u-ea644 for 18501001T0000Z/coupled looks like a red herring.
Let’s look at how long the previous equivalent tasks took by searching for instances of INFO - in the job.out file…
$ grep -a 'INFO -' /home/n02/n02/eelrm/cylc-run/u-ea644/run1/log/job/18510401T0000Z/coupled/NN/job.out
2026-07-16T04:16:46Z INFO - started
2026-07-16T05:51:34Z INFO - succeeded
This is about 1 hour 35 minutes and the same duration for the next cycle point too.
From a look around your output directory, the sea ice and atmosphere restarts for 1/1/1852 have been created but not for the ocean.
One can see that the simulation has tried to run from 18520101T0000Z from the atmosphere xhist file…
/home/n02/n02/eelrm/cylc-run/u-ea644/run1/share/data/History_Data/ea644.xhist
You can also examine the exact time steps that are attempted in ‘processor zero’ output file for each coupled task run in work/[cycle point]/coupled/pe_output/*pe000.
Cheers
Jonny
Hi Jonny,
Thanks. How should I proceed? Should I set the coupled task to succeeded?
Lauren
Hi Lauren
You shouldn’t use cylc set intervention here (the Cylc 8 equivalent of manualy setting a task’s state) because the 1/1/1852 restart files were only created for the UM and CICE models, not for NEMO. This could be for a myriad of reasons and it’s happened to me many times.
Cylc 8 ‘intervention’ documentations is here 
Interventions — Cylc 8.6.5 documentation.
For u-ea644, the latest atmosphere restarts available look to be 1/10/1851…
- Sea ice List of sea ice restart files sorted by date.
$ find . -name ea644i*restart*.nc | sort
./share/data/History_Data/CICEhist/ea644i.restart.1851-10-01-00000.nc
./share/data/History_Data/CICEhist/ea644i.restart.1852-01-01-00000.nc
- Atmosphere List of atmopshere restart files sorted by date.
$ find . -name ea644a.da* | sort
./share/data/History_Data/ea644a.da18510401_00
./share/data/History_Data/ea644a.da18510701_00
./share/data/History_Data/ea644a.da18511001_00
./share/data/History_Data/ea644a.da18520101_00
- NEMO List of ocean restart files for processing zero sorted by date (by default NEMO output is separated by processor and note that there are different restart files for the physical ocean, the tracer fields, and icebergs).
$ find . -name ea644o*restart*0000.nc | sort
./share/data/History_Data/NEMOhist/ea644o_18510701_restart_0000.nc
./share/data/History_Data/NEMOhist/ea644o_18510701_restart_trc_0000.nc
./share/data/History_Data/NEMOhist/ea644o_18511001_restart_0000.nc
./share/data/History_Data/NEMOhist/ea644o_18511001_restart_trc_0000.nc
./share/data/History_Data/NEMOhist/ea644o_icebergs_18510701_restart_0000.nc
./share/data/History_Data/NEMOhist/ea644o_icebergs_18511001_restart_0000.nc
So in summary, as before you can either start again from fresh or re-run the 18511001T0000Z cycle point.
Of course if you see exactly the same error again then this may indicate something more fundamental is wrong but this seems unlikely I’d say.
Cheers
Jonny
Hi Jonny,
I couldn’t get the 185110 cycle point to work (same timeout issue on multiple retries) but I’ve submitted a new run and it’s now successfully got past this point.
Many thanks for your help.
Lauren
Hi Lauren.
That’s great news, thanks for letting us know. 
Jonny