# Restart cylc workflow following postproc\_atmos failure **URL:** \ **Category:** Unified Model\ **Tags:** ARCHER2\ **Created:** [22 July 2026 11:01 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137 "2026-07-22T11:01:53Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![lmarshall](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@lmarshall](https://cms-helpdesk.ncas.ac.uk/u/lmarshall)\ **Post date:** [22 July 2026 11:01 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/1 "2026-07-22T11:01:54Z")
Dear CMS team, My suite u-ea645 has failed in postproc\_atmos on cycle 185210 due to a corrupt file (as in [Error at postproc\_atmos - Unified Model - NCAS Modelling Support](https://cms-helpdesk.ncas.ac.uk/t/error-at-postproc-atmos/854)). I have tried retriggering this cycle but as the other postprocessing tasks have run, the ocean and cice restart files have been removed. I am therefore trying to restart from where I have archived dumps - 1852 01. Is it possible to move these archived restarts back into the History\_data directory and retrigger the run from the 1852 01 cycle or should I do a full restart (following [tips\_CRgeneral – MOCI](https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#Restartingfromarchivedrestarts)) using cylc vip (I assume this would create a run2?) Many thanks for your help, Lauren ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [22 July 2026 12:42 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/2 "2026-07-22T12:42:16Z")
Hi Lauren. Yes you can certainly bring back your restarts and then re-trigger using `flow=none` to isolate the task(s) that need re-running. You’ll need to… 1. Edit the `ainitial`, `NEMO_START`, `NEMO_ICEBERGS` and `CICE_START` environment variables in `~/roses/u-ea645/app/{um,nemo_cice}/rose-app.conf`. 2. ‘Reinstall’ the workflow using `cylc reinstall` or `cylc vr` ( **v** alidate, **r** einstall). 3. Trigger the task(s) that need rerunning using `flow=none`. You can read about this here or check out the `cylc trigger` docs by `cylc trigger -h` (see below). Also, yes, `cylc vip` will start a `run2` from the start; VIP is short for **v** alidate, **i** nstall, **p** lay. 😃 Cheers Jonny ```auto $cylc trigger -h | head Usage: cylc trigger [OPTIONS] ID ... Trigger a group of one or more tasks, respecting dependencies among them. Prerequisites on tasks outside of the group will be satisfied automatically. Tasks will be removed if necessary to allow re-run without intervention, so triggered tasks that are preparing, submitted, or running may be killed. Tasks that lead into a group will run immediately even if the workflow is ``` ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [22 July 2026 13:39 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/3 "2026-07-22T13:39:55Z")
… also there is some information on restarting failing workflows here… [https://code.metoffice.gov.uk/trac/moci/wiki/tips\_CRgeneral#RestartingFailingSuites](https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#RestartingFailingSuites) Thanks to @RosalynHatcher for the tip! Jonny ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [22 July 2026 13:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/4 "2026-07-22T13:58:15Z")
… _and_ note that the tips in the MOSRS webpage should negate the need to change the atmosphere, ocean, etc, restarts as I noted above; changes in them are really only necessitated when starting a new run. Triggering with `flow=none` is still relevant though. Anyway, let us know how you get on! Cheers Jonny ---
**Author:** ![lmarshall](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@lmarshall](https://cms-helpdesk.ncas.ac.uk/u/lmarshall)\ **Post date:** [27 July 2026 13:12 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/5 "2026-07-27T13:12:38Z")
Hi Jonny, Thank you for the info. I’ve attempted to start from the previous January archived dumps but I’m currently getting new runs that then can’t find the executables. Could you point me to where I’ve gone wrong? Why do I need the flow=None? Or, alternatively, should I build the executables for this cycle? I also have another run (u-ea644) that is failing the coupled task due to time limits. I’ve upped the wallclock by an hour but it’s still running out of time. Is there anything else going on? Thanks, Lauren ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [27 July 2026 14:03 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/6 "2026-07-27T14:03:04Z")
Hi Lauren, Can you please change the permissions on your PUMA & ARCHER2 directories so that we can see into them? ```auto chmod -R g+rX /home/n02/n02/ chmod -R g+rX /work/n02/n02/ ``` What steps did you take to try and roll back to the January? I’m guessing not by following the [https://code.metoffice.gov.uk/trac/moci/wiki/tips\_CRgeneral#RestartingFailingSuites](https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#RestartingFailingSuites) instructions? Did you `cylc vip` to start a new run? Regarding the coupled task running out of time. We have had occurences of slow running on some of the compute nodes. I’ll take a look when I can see your files to confirm. Cheers, Ros ---
**Author:** ![lmarshall](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@lmarshall](https://cms-helpdesk.ncas.ac.uk/u/lmarshall)\ **Post date:** [27 July 2026 14:38 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/7 "2026-07-27T14:38:11Z")
Hi Ros, Thank you. I followed the instructions under ‘restarting from archived restarts’ up until point 10: run as NRUN. I have then tried both cylc vr for the original run1 and retriggered the 1852-01 cycle, and cylc vips - run 2 I hadn’t turned off the build and recon, run3 latest attempt. Thanks a lot, Lauren ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [27 July 2026 15:34 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/8 "2026-07-27T15:34:23Z")
Hi Lauren, If you’ve got all the18521001 restart dumps I’m pretty sure we should be able to patch run1 up and then just retrigger the 18521001T0000Z/coupled task. But if you’ve made changes to the suite now it’s probably going to be best to continue along the track you’ve now started. With Cylc 8 you either have to turn the build on or manually add in a symlink to it, since a new run uses a completely new run directory and knows nothing about any previous runs. To go the symlink root, on ARCHER2 ```auto cd /work/n02/n02/eelrm/cylc-run/u-ea645/run3/share ln -s ../../run1/share/fcm_make_um fcm_make_um ln -s ../../run1/share/fcm_make_ocean fcm_make_ocean ``` Then retrigger the coupled task. Cheers, Ros ---
**Author:** ![lmarshall](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@lmarshall](https://cms-helpdesk.ncas.ac.uk/u/lmarshall)\ **Post date:** [28 July 2026 08:05 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/9 "2026-07-28T08:05:06Z")
Hi Ros, Thanks. I’ve given the symlink option a go but am now getting a Mismatch between model\_basis\_time read from namelist and validity time read. Where is the 1850 date coming from? Thanks, Lauren ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [28 July 2026 09:51 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/10 "2026-07-28T09:51:09Z")
HI Lauren. I’m just doing a test of this setup. The date mismatch error will presumably be because of the lack of the `recon` step which, in the ‘normal’ case overwrites the dump’s date (1/1/3075) with the required workflow `BASIS` time. Cheers Jonny ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [28 July 2026 12:25 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/11 "2026-07-28T12:25:12Z")
Hi Lauren. I’ve done some digging and certainly a cause of your issue that is causing your date-time mismatch is that your restart file `/work/n02/n02/eelrm/Dumps/forcevol/ea645a.da18520101_00` appears to be erroneous. From inspecting this file and comparing its output (I’ve looked at QCF and U wind) it is identical to `/work/n02/n02/eelrm/Dumps/forcevol/by230a.da30750101_00` and has the 1/1/3075 date stamp. What I think has happened is that the reconfiguration step was run (replacing the `astart` parameter in `~/roses/u-ea645/app/um/rose-app.conf` with the `ea645a.da18520101_00` path above. This will (quietly) overwrite a file with the same name in the same directory. My advice now would be to either… 1. Get back the archived restart for the atmosphere and try again. 2. Run a fresh simulation from `18500101T0000Z` since you are so close to the beginning of the run. Hope that helps. Jonny ---
**Author:** ![lmarshall](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@lmarshall](https://cms-helpdesk.ncas.ac.uk/u/lmarshall)\ **Post date:** [28 July 2026 12:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/12 "2026-07-28T12:55:54Z")
Hi Jonny, Thanks a lot - makes sense. I’ve reverted the suite and submitted a fresh simulation. Are there any other updates on the wallclock issue for u-ea644? Many thanks, Lauren ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [28 July 2026 14:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/13 "2026-07-28T14:44:09Z")
Hi Lauren. The wallclock issue with `u-ea644` for `18501001T0000Z/coupled` looks like a red herring. Let’s look at how long the previous equivalent tasks took by searching for instances of `INFO -` in the `job.out` file… ```auto $ grep -a 'INFO -' /home/n02/n02/eelrm/cylc-run/u-ea644/run1/log/job/18510401T0000Z/coupled/NN/job.out 2026-07-16T04:16:46Z INFO - started 2026-07-16T05:51:34Z INFO - succeeded ``` This is about 1 hour 35 minutes and the same duration for the next cycle point too. From a look around your output directory, the sea ice and atmosphere restarts for 1/1/1852 have been created but not for the ocean. One can see that the simulation has tried to run from `18520101T0000Z` from the atmosphere `xhist` file… ```auto /home/n02/n02/eelrm/cylc-run/u-ea644/run1/share/data/History_Data/ea644.xhist ``` You can also examine the exact time steps that are attempted in ‘processor zero’ output file for each coupled task run in `work/[cycle point]/coupled/pe_output/*pe000`. Cheers Jonny ---
**Author:** ![lmarshall](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@lmarshall](https://cms-helpdesk.ncas.ac.uk/u/lmarshall)\ **Post date:** [29 July 2026 08:47 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/14 "2026-07-29T08:47:19Z")
Hi Jonny, Thanks. How should I proceed? Should I set the coupled task to succeeded? Lauren ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [29 July 2026 10:48 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/15 "2026-07-29T10:48:24Z")
Hi Lauren You shouldn’t use `cylc set` intervention here (the Cylc 8 equivalent of manualy setting a task’s state) because the 1/1/1852 restart files were only created for the UM and CICE models, not for NEMO. This could be for a myriad of reasons and it’s happened to me many times. Cylc 8 ‘intervention’ documentations is here ➡ [Interventions — Cylc 8.6.5 documentation](https://cylc.github.io/cylc-doc/stable/html/user-guide/interventions/index.html). For `u-ea644`, the latest atmosphere restarts available look to be 1/10/1851… - **Sea ice** ➡ List of sea ice restart files sorted by date. ```auto $ find . -name ea644i*restart*.nc | sort ./share/data/History_Data/CICEhist/ea644i.restart.1851-10-01-00000.nc ./share/data/History_Data/CICEhist/ea644i.restart.1852-01-01-00000.nc ``` - **Atmosphere** ➡ List of atmopshere restart files sorted by date. ```auto $ find . -name ea644a.da* | sort ./share/data/History_Data/ea644a.da18510401_00 ./share/data/History_Data/ea644a.da18510701_00 ./share/data/History_Data/ea644a.da18511001_00 ./share/data/History_Data/ea644a.da18520101_00 ``` - **NEMO** ➡ List of ocean restart files for processing zero sorted by date (by default NEMO output is separated by processor and note that there are different restart files for the physical ocean, the tracer fields, and icebergs). ```auto $ find . -name ea644o*restart*0000.nc | sort ./share/data/History_Data/NEMOhist/ea644o_18510701_restart_0000.nc ./share/data/History_Data/NEMOhist/ea644o_18510701_restart_trc_0000.nc ./share/data/History_Data/NEMOhist/ea644o_18511001_restart_0000.nc ./share/data/History_Data/NEMOhist/ea644o_18511001_restart_trc_0000.nc ./share/data/History_Data/NEMOhist/ea644o_icebergs_18510701_restart_0000.nc ./share/data/History_Data/NEMOhist/ea644o_icebergs_18511001_restart_0000.nc ``` So in summary, as before you can either start again from fresh _or_ re-run the `18511001T0000Z` cycle point. Of course if you see exactly the same error again then this may indicate something more fundamental is wrong but this seems unlikely I’d say. Cheers Jonny ---
**Author:** ![lmarshall](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@lmarshall](https://cms-helpdesk.ncas.ac.uk/u/lmarshall)\ **Post date:** [30 July 2026 12:46 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/16 "2026-07-30T12:46:10Z")
Hi Jonny, I couldn’t get the 185110 cycle point to work (same timeout issue on multiple retries) but I’ve submitted a new run and it’s now successfully got past this point. Many thanks for your help. Lauren ---
**Author:** ![jonnyhtw](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/jonnyhtw/32/548_2.png) [@jonnyhtw](https://cms-helpdesk.ncas.ac.uk/u/jonnyhtw)\ **Post date:** [30 July 2026 13:02 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/17 "2026-07-30T13:02:10Z")
Hi Lauren. That’s great news, thanks for letting us know. 🥳 Jonny ---
**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/1fd2411499ffcbc299fe756cd5cdf26e44956558.png) [@system](https://cms-helpdesk.ncas.ac.uk/u/system)\ **Post date:** [29 August 2026 13:02 UTC](https://cms-helpdesk.ncas.ac.uk/t/restart-cylc-workflow-following-postproc-atmos-failure/2137/18 "2026-08-29T13:02:43Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.