# Cycle point for restarting suites? **URL:** \ **Category:** Unified Model\ **Tags:** ARCHER2, PUMATest\ **Created:** [17 February 2022 14:17 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476 "2022-02-17T14:17:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [17 February 2022 14:17 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/1 "2022-02-17T14:17:43Z")
Hi, I’m trying to follow the instructions here to restart HadGEM3 suites ([http://cms.ncas.ac.uk/wiki/pumatest-transition](http://cms.ncas.ac.uk/wiki/pumatest-transition)), but all suites fail at the coupled step. Could this be because restart files do not exist for this cycle point/the warm start is pointing to the wrong restart files for this cycle point? 1. I first tried to start from the last completed month in the ARCHER2 /work/n02/n02/radiam24/archive/u-exptid, but this failed in the coupled step and I got errors in the model output files - e.g. in ocean.output from NEMO, there was **===\>\>\> : E R R O R** \*\* ===========\*\* \*\* ===\>\>\>\> : problem with nittrc000 for the restart\*\* \*\* verify the restart file or rerun with nn\_rsttr = 0 (namelist)\*\* \*\* ===\>\>\> : E R R O R\*\* \*\* ===========\*\* \*\* STOP\*\* \*\* Critical errors in NEMO initialisation\*\* \*\* huge E-R-R-O-R : immediate stop\*\* and in output from the UM, there were a lot of MPICH errors. 1. I then tried to restart from the cycle point the model was at when PUMA went down, but got the following error: **TypeError: not enough arguments for format string** **[FAIL] run\_model \<\<'STDIN’** \*\*[FAIL] \*\* **[FAIL] ‘STDIN’ # return-code=1** **2022-02-17T13:54:15Z CRITICAL - failed/EXIT** So will I need to restart from the last cycle point I know has a complete set of restart dumps? This is a bit inconvenient since I only have restart dumps from January and December each year. Best wishes, Rachel ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [17 February 2022 14:31 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/2 "2022-02-17T14:31:28Z")
Hi Rachel, What’s the suite id? And please post which cycle points you tried to restart from. You should be able to see from the log files on ARCHER2 where the last cycle had got to in order to work out which cycle you needed to start from. Cheers, Ros. ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [17 February 2022 14:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/3 "2022-02-17T14:55:38Z")
Hi Ros, I tried | suite | 1. Cycle point: last archived on ARCHER2 | 2. Cycle point: running when PUMA went down | | --- | --- | --- | | u-cl809 | 18500201T0000Z | 18500301T0000Z | | u-ck767 | 18700701T0000Z | 18700801T0000Z | | u-ck831 | 18620301T0000Z | 18620401T0000Z | The first set of cycle points all gave the first error in the last post, and the second set of cycle points all gave the second error. I also noticed the errors are similar to those here: [Issues with a warm start using pumatest - #4 by aschurer](https://cms-helpdesk.ncas.ac.uk/t/issues-with-a-warm-start-using-pumatest/475/4). Rachel ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [18 February 2022 08:45 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/4 "2022-02-18T08:45:48Z")
Hi Rachel, The cycle points that were running when PUMA went down was where you needed to restart from. Having tried to back up even further has confused it - not entirely sure what happened. I’ve only looked at u-cl809 so far. If you look in the job.out file you’ll see more informative errors that indicate the UM, NEMO & CICE restarts are now out of sync. ```auto [INFO] The NEMO restart data does not match the current cycle time . Cycle time is 18500301 NEMO restart time is 18500201 [INFO] Remove any NEMO dumps currently ahead of the current cycletime, and pick up the dump at this time ``` So for u-cl809 you will need to follow the instructions at [https://code.metoffice.gov.uk/trac/moci/wiki/tips\_CRgeneral#RestartingFailingSuites](https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#RestartingFailingSuites) to sort out the restarts and rerun from 18500201. I will look at the other 2 suites and check they are in the same situation. Regards, Ros. ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [19 February 2022 12:56 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/5 "2022-02-19T12:56:55Z")
Hi Ros, Thanks so much for clarifying. I tried this (and followed the instructions very carefully so the timestamps of the required start files did match up), but the suite failed anyway with an initialisation error. (I can’t be more specific about the error, sorry, as I didn’t make a note of it before trying to start the run again and I think RoseBush is still down). Since it was only 2 months into the run, I then tried to start the run over again from the beginning, but this also failed (which I should have realised it would do as the cylc-run directory hasn’t been copied over to PUMATest). I have saved in another directory most, but not all, of the data from the History\_Data file for this suite before PUMA went down, and I do have access to the previous .xhist files for this suite. So is it worth trying to restart this suite as well as my other previously running suites, given they will probably fail with the same initialisation error? Or is there another way of restarting suites I could try? Best wishes, Rachel ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [19 February 2022 14:34 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/6 "2022-02-19T14:34:55Z")
Hi Rachel, As you’ve tried to restart from beginning again I won’t be able to see what went wrong after following those Met Office instructions. If it’s only 2 months in, it will be easiest just to start the suite again from the beginning. `rose suite-run --new` to give a clean run. The other suites look further into the run. So worth trying to go through those instructions on u-ck767 and see if it the warm start then works ok. If it doesn’t please let us know so we can take a look - don’t try anything else with it otherwise it makes it very difficult to see what’s going on. Rose bush is on pumatest. The url just changed slightly inline with the hostname change: [https://pumatest.nerc.ac.uk/rose-bush/](https://pumatest.nerc.ac.uk/rose-bush/) Cheers, Ros. ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [21 February 2022 08:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/7 "2022-02-21T08:14:08Z")
P.S. In your `~/.profile` please change the `SCRATCH` directory to be: `export SCRATCH=/export/puma/data-01/scratch/$USER` ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [21 February 2022 11:47 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/8 "2022-02-21T11:47:25Z")
Hi Rachel, In the `site/archer2.rc` file in the suites. In the `[[EXTRACT_RESOURCE]]` section can you please comment (#) out the line: `# script = "rose task-run --verbose --define=fast-dest-root-orig=$SCRATCH --define='args=--archive --ignore-lock'"` So it does not use the scratch disk. Thanks. Cheers, Ros. ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [21 February 2022 13:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/9 "2022-02-21T13:55:30Z")
Hi Ros, Thanks for the help. I’ve now started u-cl809 from the beginning, and it is running fine. Should I also comment out the # script = “rose task-run --verbose --define=fast-dest-root-orig=$SCRATCH --define=‘args=–archive --ignore-lock’” for this suite, and reload it? I am now trying to restart u-ck767 from 18700701T000Z, and followed the MetOffice instructions to get consistent datestamps that you shared. I then tried to restart it using: rose suite-run – --warm 18700701T0000Z This suite then failed in the coupled step with a MPICH error in job.err, preceded by the same warning I previously got with u-cl809, which looks like: WARNING ??? ? Warning code: -10 ? Warning from routine: INIT\_PP\_CRUN ? Warning message: Error: Failed to load fixed header. ? Warning from processor: 0 ? Warning number: 34 Could you please let me know what to try next? Best wishes, Rachel ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [21 February 2022 14:43 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/10 "2022-02-21T14:43:09Z")
Hi Rachel Try moving` /home/n02/n02/radiam24/cylc-run/u-ck767/work/18700801T0000Z` away (it should be OK to delete it), then warm start 18700701T0000Z again. Grenville ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [22 February 2022 15:48 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/11 "2022-02-22T15:48:13Z")
Hi, I tried that but get the same error. Best wishes, Rachel ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [23 February 2022 10:25 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/12 "2022-02-23T10:25:51Z")
Hi, Is there anything else I should try? Or should I make a new suite with the same model configuration as I used for u-ck767, and initialise the run from the last set of restart dumps I have for u-ck767? Best wishes, Rachel ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [23 February 2022 10:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/13 "2022-02-23T10:57:07Z")
Hi Rachel, Think this error is due to an incorrect namelist file please try copying in the nemo namelist from the end of the previous cycle: `archer2$ cp ~/cylc-run/u-ck767/work/18700601T0000Z/coupled/namelist_cfg ~/cylc-run/u-ck767/share/data/History_Data/NEMOhist/namelist_cfg` Then warm start from 18700701T0000Z again. Regards, Ros For info error in job.err is a generic MPI error and more detailed error in ocean.output is: ```auto ===>>> : E R R O R =========== ===>>>> : problem with nittrc000 for the restart verify the restart file or rerun with nn_rsttr = 0 (namelist) ``` ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [23 February 2022 13:27 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/14 "2022-02-23T13:27:20Z")
Hi Ros, Thanks, I retriggered and ran the 18700701 coupled task. The run now fails in the postproc\_atmos task with the same error as here (11/15 on [Issues with a warm start using pumatest - #11 by aschurer](https://cms-helpdesk.ncas.ac.uk/t/issues-with-a-warm-start-using-pumatest/475/11)) I checked in the 18700701 History\_Data directory and it contains old files of 0 length from my last attempt at running the task (2 days ago): -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p11870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p21870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p31870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p41870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p51870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p618700621 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p718700621 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p818700621 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.p918700621 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pa1870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pb18700401 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pc18700401 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pd1870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pe1870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pf18700401 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pg18700401 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.ph1870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pi18700401 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pj18700401 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pk1870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pl1870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pn18700621 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pt18700621 -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pu1870jun -rw-r–r-- 1 radiam24 n02 0 Feb 21 13:29 ck767a.pv1870jun Am I ok to delete the old 0-length files and retrigger the postproc\_atmos task? Or do I need to delete a lot more of the files in History\_Data and retrigger from the coupled task? Best wishes, Rachel ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [23 February 2022 13:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/15 "2022-02-23T13:44:05Z")
Hi Rachel, That’s good to hear it started running ok again. Yes, delete those zero-length files in the History\_Data directory and retrigger the postproc\_atmos task. Cheers, Ros. ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [23 February 2022 16:03 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/16 "2022-02-23T16:03:40Z")
Hi Ros, Thanks, u-ck767 is running fine now. However, I then tried to run u-ck831 from cycle point 18620301T0000Z after applying all the same fixes because it previously failed with the same error as u-ck767 (I also used cp ~/cylc-run/u-ck831/work/18620201T0000Z/coupled/namelist\_cfg ~/cylc-run/u-ck831/share/data/History\_Data/NEMOhist/namelist\_cfg; 18620201T0000Z completed successfully on ARCHER2 before PUMA went down). u-ck831 failed in the coupled step: **Atm\_Step: Timestep 315378 Model time: 1862-03-01 06:00:00** **update\_dpsidt: updating based on existing values** **update\_pattern: updating coeffc and coeffs** **ERROR detected in routine STWORK** **: no. of output fields (=12721) exceeds no. of reserved PP headers for unit 28** **STWORK: Error when processing diagnostic section 3, item 227, code 4** **???** **???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!** **? Error code: 4** **? Error from routine: STWORK** **? Error message: STWORK: Number of fields exceeds reserved headers for unit 28** **? Error from processor: 0** **? Error number: 39** **???** I could increase the number of headers, but cycle 18620301T0000Z ran successfully before PUMA went down, so is the real problem because of something else? Rachel P.S. Is it that the History\_Data directory contains data from previous runs that may be confusing the model? In pe\_output/ck831.fort6.000: **IO: Open: /work/n02/n02/radiam24/cylc-run/u-ck831/share/data/History\_Data/ck831a.pa1862feb on unit 11 (Server = 11)** **loadHeader: The file on unit 11 does not have enough content to load the header** **???** **??? WARNING ???** **? Warning code: -10** **? Warning from routine: INIT\_PP\_CRUN** **? Warning message: Error: Failed to load fixed header.** **? Warning from processor: 0** **? Warning number: 13** I checked and /work/n02/n02/radiam24/cylc-run/u-ck831/share/data/History\_Data/ck831a.pa1862feb is one of several zero-length files created just now by the coupled task. ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [24 February 2022 16:16 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/17 "2022-02-24T16:16:28Z")
Rachel The error is coming from the atmosphere model - it has nothing to do with restarting the model. I can’t see output for a previous run of cycle 18620301T0000Z, but I can only suppose that the previous run did not get as far as time step 315378. I’d not worry about the warning - the pa file re-initialises each month and ck831a.pa1862feb.pp is in the archive directory. post proc will complain about the empty files 'though Grenville. ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [25 February 2022 10:50 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/18 "2022-02-25T10:50:47Z")
Hi Grenville, Thanks. Where would I find which stream is attached to unit 28? I can’t find any reference to unit 28 in /work/n02/n02/radiam24/cylc-run/u-ck831/log/job/18620301T0000Z/coupled/01/job.out. Best wishes, Rachel ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [25 February 2022 11:08 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/19 "2022-02-25T11:08:45Z")
Hi Rachel, Sometimes suites are not set up to put the `pe0` output into the `job.out` file. Have a look at: `/work/n02/n02/radiam24/cylc-run/u-ck831/work/18620301T0000Z/coupled/pe_output/ck831.fort6.pe000` ```auto FILE_MANAGER: Assigned : /work/n02/n02/radiam24/cylc-run/u-ck831/share/data/History_Data/ck831a.pc18620101 FILE_MANAGER: : id : pp2 FILE_MANAGER: : Unit : 28 (portio) ``` Cheers, Ros. ---
**Author:** ![racheldiamond](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@racheldiamond](https://cms-helpdesk.ncas.ac.uk/u/racheldiamond)\ **Post date:** [25 February 2022 13:10 UTC](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476/20 "2022-02-25T13:10:45Z")
Hi Ros, Thanks. I changed reinit\_step from 90 to 60 for pp2 (and pp5 and pp9), which fixed that problem. Do I need to change reinit\_step for all the other streams as well, to be consistent? However, the coupled step still fails with new error in job.err, that looks a bit like this segmentation fault [http://cms.ncas.ac.uk/ticket/2429?cversion=0&cnum\_hist=1](http://cms.ncas.ac.uk/ticket/2429?cversion=0&cnum_hist=1) The error I get is - [0] exceptions: Program is “/mnt/lustre/a2fs-work2/work/n02/n02/radiam24/cylc-run/u-ck831/work/18620301T0000Z/coupled/./atmos.exe” Warning in umPrintMgr: umPrintExceptionHandler : Handler Invoked [0] exceptions: Data address (si\_addr): 0x00000000; rip: 0x005963ea [0] exceptions: [backtrace]: has 14 elements: [0] exceptions: [backtrace]: ( 1) : Address: [0x005963ea] [0] exceptions: [backtrace]: ( 1) : stwork\_ (\* Cannot Locate _) [0] exceptions: [backtrace]: ( 2) : Address: [0x00416df4] [0] exceptions: [backtrace]: ( 2) : signal\_do\_backtrace\_linux in file /mnt/lustre/a2fs-work2/work/n02/n02/radiam24/cylc-run/u-ck831/share/fcm\_make\_um/extract/um/src/control/c\_code/exceptions/exceptions-platform/exceptions-linux.c line 81 [0] exceptions: [backtrace]: ( 3) : Address: [0x00415b88] [0] exceptions: [backtrace]: ( 3) : signal\_do\_backtrace in file /mnt/lustre/a2fs-work2/work/n02/n02/radiam24/cylc-run/u-ck831/share/fcm\_make\_um/extract/um/src/control/c\_code/exceptions/exceptions.c line 81 [0] exceptions: [backtrace]: ( 4) : Address: [0x2b84e0a3c2d0] [0] exceptions: [backtrace]: ( 4) : ?? (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 5) : Address: [0x005963ea] [0] exceptions: [backtrace]: ( 5) : stwork\_ (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 6) : Address: [0x005926b4] [0] exceptions: [backtrace]: ( 6) : stash\_ (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 7) : Address: [0x0097eeca] [0] exceptions: [backtrace]: ( 7) : st\_diag1\_ (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 8) : Address: [0x00ac5187] [0] exceptions: [backtrace]: ( 8) : atm\_step\_4a\_ (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 9) : Address: [0x00447f4e] [0] exceptions: [backtrace]: ( 9) : u\_model\_4a\_ (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 10) : Address: [0x0040fc08] [0] exceptions: [backtrace]: ( 10) : um\_shell\_ (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 11) : Address: [0x00408698] [0] exceptions: [backtrace]: ( 11) : main (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 12) : Address: [0x00408698] [0] exceptions: [backtrace]: ( 12) : main (_ Cannot Locate _) [0] exceptions: [backtrace]: ( 13) : Address: [0x2b84e0c6c34a] [0] exceptions: [backtrace]: ( 13) : ?? (_ Cannot Locate \*) [0] exceptions: [backtrace]: ( 14) : Address: [0x0040849a] [0] exceptions: [backtrace]: ( 14) : \_start in file /home/abuild/rpmbuild/BUILD/glibc-2.26/csu/…/sysdeps/x86\_64/start.S line 122 [0] exceptions: [0] exceptions: To find the source line for an entry in the backtrace; [0] exceptions: run addr2line --exe=\ [0] exceptions: where address is given as [0x] above [0] exceptions: srun: error: nid002159: task 0: Exited with exit code 11 srun: launch/slurm: \_step\_signal: Terminating StepId=1158426.0+0 srun: launch/slurm: \_step\_signal: Terminating StepId=1158426.0+1 slurmstepd: error: \*\*\* STEP 1158426.0+1 ON nid005786 CANCELLED AT 2022-02-25T12:56:10 \*\*\* slurmstepd: error: \*\*\* STEP 1158426.0+0 ON nid002159 CANCELLED AT 2022-02-25T12:56:10 \*\*\* slurmstepd: error: \*\*\* STEP 1158426.0+2 ON nid005787 CANCELLED AT 2022-02-25T12:56:10 \*\*\* srun: launch/slurm: \_step\_signal: Terminating StepId=1158426.0+2 srun: error: nid005787: tasks 0-1: Terminated srun: error: nid005840: tasks 2-3: Terminated srun: error: nid005786: task 7: Terminated srun: error: nid002160: tasks 50-99: Terminated srun: error: nid002159: tasks 1-49: Terminated srun: error: nid002161: tasks 100-148: Terminated srun: error: nid005860: tasks 4-5: Terminated srun: Force Terminated StepId=1158426.0+2 srun: error: nid005785: tasks 149-197: Terminated srun: Force Terminated StepId=1158426.0+0 srun: error: nid005786: tasks 0-6,8-127: Terminated srun: Force Terminated StepId=1158426.0+1 [FAIL] run\_model \<\<‘ **STDIN** ’ [FAIL] [FAIL] ‘ **STDIN** ’ # return-code=143 2022-02-25T12:56:11Z CRITICAL - failed/EXIT Rachel [Next page](https://cms-helpdesk.ncas.ac.uk/t/cycle-point-for-restarting-suites/476.md?page=2)