# Cancelled due to time limit \*\*\*

**URL:** <https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084>\
**Category:** Uncategorized\
**Tags:** ARCHER2\
**Created:** [1 August 2023 08:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084 "2023-08-01T08:57:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [1 August 2023 08:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/1 "2023-08-01T08:57:21Z")

</div>

I am getting the following error trying to run u-cy520  
(fails at coupled)

- a previous post suggest going into /site/archer2.rc - but there seems to be a number of possible lines with ‘execution time limit’?

Many thanks,  
Jeremy

???  
??? WARNING ???  
? Warning code: -1  
? Warning from routine: ASAD\_CINIT  
? Warning message: NRSTEPS IS OUT OF RANGE, RESETTING  
? Warning from processor: 0  
? Warning number: 68  
???

slurmstepd: error: \*\*\* STEP 4097245.0+0 ON nid002216 CANCELLED AT 2023-08-01T00:30:49 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* JOB 4097245 ON nid002216 CANCELLED AT 2023-08-01T00:30:49 DUE TO TIME LIMIT \*\*\*  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
slurmstepd: error: \*\*\* STEP 4097245.0+2 ON nid004049 CANCELLED AT 2023-08-01T00:30:49 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* STEP 4097245.0+1 ON nid004034 CANCELLED AT 2023-08-01T00:30:49 DUE TO TIME LIMIT \*\*\*  
srun: error: nid003341: tasks 704-711,713-746,748-767: Terminated  
srun: launch/slurm: \_step\_signal: Terminating StepId=4097245.0+0  
2023-07-31T23:30:50Z CRITICAL - failed/EXIT

---

<div class="post-metadata">

**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\
**Post date:** [1 August 2023 09:10 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/2 "2023-08-01T09:10:12Z")

</div>

Hi Jeremy,

In rose edit GUI go to _suite conf → run initialisation and cycling_ and increase the wallclock time.

Regards  
Ros.

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [1 August 2023 12:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/3 "2023-08-01T12:15:24Z")

</div>

Thanks Ros,

I changed to 4hr 30 mins - I am now getting fail at UMbuild \> fcm\_make2\_um

Many thanks for help,

activity log:  
[jobs-submit ret\_code] 0  
[jobs-submit out] 2023-08-01T09:18:28Z|19501001T0000Z/fcm\_make2\_um/01|0|4102177  
([jgrist02@login.archer2.ac.uk](mailto:jgrist02@login.archer2.ac.uk)) 2023-08-01T09:18:28Z [STDOUT] Submitted batch job 4102177  
[jobs-poll ret\_code] 0  
[jobs-poll out] 2023-08-01T09:23:30Z|19501001T0000Z/fcm\_make2\_um/01|{“batch\_sys\_name”: “slurm”, “batch\_sys\_job\_id”: “4102177”, “batch\_sys\_exit\_polled”: 0, “time\_submit\_exit”: “2023-08-01T09:18:28Z”, “time\_run”: “2023-08-01T09:18:42Z”}  
[jobs-poll ret\_code] 0  
[jobs-poll out] 2023-08-01T09:28:33Z|19501001T0000Z/fcm\_make2\_um/01|{“batch\_sys\_name”: “slurm”, “batch\_sys\_job\_id”: “4102177”, “batch\_sys\_exit\_polled”: 0, “time\_submit\_exit”: “2023-08-01T09:18:28Z”, “time\_run”: “2023-08-01T09:18:42Z”}  
[jobs-poll ret\_code] 0  
[jobs-poll out] 2023-08-01T09:33:33Z|19501001T0000Z/fcm\_make2\_um/01|{“batch\_sys\_name”: “slurm”, “batch\_sys\_job\_id”: “4102177”, “batch\_sys\_exit\_polled”: 0, “time\_submit\_exit”: “2023-08-01T09:18:28Z”, “time\_run”: “2023-08-01T09:18:42Z”}  
[jobs-poll ret\_code] 0  
[jobs-poll out] 2023-08-01T09:38:35Z|19501001T0000Z/fcm\_make2\_um/01|{“batch\_sys\_name”: “slurm”, “batch\_sys\_job\_id”: “4102177”, “batch\_sys\_exit\_polled”: 0, “time\_submit\_exit”: “2023-08-01T09:18:28Z”, “time\_run”: “2023-08-01T09:18:42Z”}  
[jobs-poll ret\_code] 0  
[jobs-poll out] 2023-08-01T09:43:35Z|19501001T0000Z/fcm\_make2\_um/01|{“batch\_sys\_name”: “slurm”, “batch\_sys\_job\_id”: “4102177”, “batch\_sys\_exit\_polled”: 0, “time\_submit\_exit”: “2023-08-01T09:18:28Z”, “time\_run”: “2023-08-01T09:18:42Z”}  
[jobs-poll ret\_code] 0  
[jobs-poll out] 2023-08-01T09:48:36Z|19501001T0000Z/fcm\_make2\_um/01|{“batch\_sys\_name”: “slurm”, “batch\_sys\_job\_id”: “4102177”, “batch\_sys\_exit\_polled”: 0, “time\_submit\_exit”: “2023-08-01T09:18:28Z”, “time\_run”: “2023-08-01T09:18:42Z”}  
[jobs-poll ret\_code] 0  
[jobs-poll out] 2023-08-01T09:53:36Z|19501001T0000Z/fcm\_make2\_um/01|{“batch\_sys\_name”: “slurm”, “batch\_sys\_job\_id”: “4102177”, “run\_status”: 1, “run\_signal”: “EXIT”, “time\_submit\_exit”: “2023-08-01T09:18:28Z”, “time\_run”: “2023-08-01T09:18:42Z”, “time\_run\_exit”: “2023-08-01T09:49:32Z”}  
[((‘job-logs-retrieve’, ‘failed’), 1) ret\_code] 0

---

<div class="post-metadata">

**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\
**Post date:** [1 August 2023 13:07 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/4 "2023-08-01T13:07:00Z")

</div>

Hi Jeremy,

If you look in the job.err file for the fcm\_make2\_um you’ll see that the compile has now failed due to timelimit.

In the site/archer2.rc file in the [[HPC\_SERIAL]] section increase the execution time limit and try again.

Just to note for the future: when you increased the timelimit for the coupled task you didn’t need to rerun the compile tasks. Just do a reload and retrigger the coupled task.

Cheers,  
Ros.

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [1 August 2023 14:31 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/5 "2023-08-01T14:31:54Z")

</div>

Hi Ros,

Thanks for your help,  
I increased the time from 30M to 60M but now get the following error:

Jeremy

[FAIL] /work/n02/n02/jgrist02/cylc-run/u-cy520/share/fcm\_make\_um/fcm-make2.lock: lock exists at the destination

[FAIL] fcm make -C /work/n02/n02/jgrist02/cylc-run/u-cy520/share/fcm\_make\_um -n 2 -j 6 # return-code=1

2023-08-01T13:37:10Z CRITICAL - failed/EXIT

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [1 August 2023 15:04 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/6 "2023-08-01T15:04:10Z")

</div>

delete` /work/n02/n02/jgrist02/cylc-run/u-cy520/share/fcm_make_um/fcm-make2.lock` (it is a directory) & retrigger

Grenville

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [2 August 2023 13:48 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/7 "2023-08-02T13:48:40Z")

</div>

Hi Grenville,

It now gets through to ‘Coupled’ where is sits ‘running’ for some time before failing

It seems to be a time out error again.

I have wall clock time set for 4h 30 min and execution time limit in the [[HPC\_SERIAL]] section of site/archer2.rc set as PT60M.

Should I increase further? Many thanks for your help, Jeremy

Snippet below from /work/n02/n02/jgrist02/cylc-run/u-cy520/log/job/19501001T0000Z/coupled/01/job.err

??? WARNING ???  
? Warning code: -1  
? Warning from routine: ASAD\_CINIT  
? Warning message: NRSTEPS IS OUT OF RANGE, RESETTING  
? Warning from processor: 0  
? Warning number: 68  
???

slurmstepd: error: \*\*\* STEP 4104686.0+0 ON nid005643 CANCELLED AT 2023-08-01T21:56:57 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* JOB 4104686 ON nid005643 CANCELLED AT 2023-08-01T21:56:57 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* STEP 4104686.0+1 ON nid005689 CANCELLED AT 2023-08-01T21:56:57 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* STEP 4104686.0+2 ON nid005721 CANCELLED AT 2023-08-01T21:56:57 DUE TO TIME LIMIT \*\*\*  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: error: nid005672: tasks 192-200,202-228,230-255: Terminated  
srun: launch/slurm: \_step\_signal: Terminating StepId=4104686.0+0  
srun: launch/slurm: \_step\_signal: Terminating StepId=4104686.0+1

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [2 August 2023 14:35 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/8 "2023-08-02T14:35:59Z")

</div>

Hi Jeremy

ROSE\_LAUNCHER\_PREOPTS\_NEMO is not quite right - it needs the `--hint=nomultithread --distribution=block:block` entries (as in ROSE\_LAUNCHER\_PREOPTS\_XIOS)

See [Updating a UM suite after the ARCHER2 O/S upgrade](https://cms.ncas.ac.uk/archer2/os-upgrade/) Coupled Suites item 3.

```auto
ROSE_LAUNCHER_PREOPTS_NEMO="--het-group=1 --nodes=9 --ntasks=1152 --tasks-per-node=128 --cpus-per-task=1 --cpu-bind=cores --export=all,OMP_NUM_THREADS=1,HYPERTHREADS=1"
ROSE_LAUNCHER_PREOPTS_XIOS="--het-group=2 --nodes=5 --ntasks=20 --tasks-per-node=4 --cpus-per-task=1 --cpu-bind=cores --hint=nomultithread --distribution=block:block --export=all,OMP_NUM_THREADS=1,HYPERTHREADS=1"

```

Grenville

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [4 August 2023 11:11 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/9 "2023-08-04T11:11:00Z")

</div>

Hi -

Thank you for your help,

I have corrected the archer2.rc file as suggested, but still get this error.  
/work/n02/n02/jgrist02/cylc-run/u-cy520/log/job/19501001T0000Z/coupled/01

???  
??? WARNING ???  
? Warning code: -1  
? Warning from routine: ASAD\_CINIT  
? Warning message: NRSTEPS IS OUT OF RANGE, RESETTING  
? Warning from processor: 0  
? Warning number: 68  
???

slurmstepd: error: \*\*\* STEP 4120259.0+0 ON nid002201 CANCELLED AT 2023-08-03T19:15:20 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* JOB 4120259 ON nid002201 CANCELLED AT 2023-08-03T19:15:20 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* STEP 4120259.0+2 ON nid004612 CANCELLED AT 2023-08-03T19:15:20 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* STEP 4120259.0+1 ON nid004603 CANCELLED AT 2023-08-03T19:15:20 DUE TO TIME LIMIT \*\*\*  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: error: nid004615: task 15: Terminated  
srun: launch/slurm: \_step\_signal: Terminating StepId=4120259.0+0  
srun: launch/slurm: \_step\_signal: Terminating StepId=4120259.0+1

As an alternative I have tried running  
u-cy010 GC5 N216 ORCA025  
As that has been updated and tested. However, this fails immediately at fcm\_make\_ocean and fcm\_make\_um

jgrist02@ln01:/work/n02/n02/jgrist02/cylc-run/u-cy621/log/job/19780901T0000Z/fcm\_make2\_pp/01\> vi job.err

Use of uninitialized value in concatenation (.) or string at /mnt/lustre/a2fs-work1/work/y07/shared/umshared/software/fcm-2019.09.0/bin/…/lib/FCM/Util.pm line 281.

Jeremy

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [4 August 2023 12:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/10 "2023-08-04T12:15:40Z")

</div>

Jeremy

Our CANARI suites have  
`--het-group=2 --nodes=7 --ntasks=28 --tasks-per-node=4 --cpus-per-task=32 --hint=nomultithread --distribution=block:block --export=all,OMP_NUM_THREADS=1,HYPERTHREADS=1 ./xios.exe`

your suite has  
-`-het-group=2 --nodes=5 --ntasks=20 --tasks-per-node=4 --cpus-per-task=1 --cpu-bind=cores --hint=nomultithread --distribution=block:block --export=all,OMP_NUM_THREADS=1,HYPERTHREADS=1 ./xios.exe`

Try setting `--cpus-per-task=32` in `ROSE_LAUNCHER_PREOPTS_XIOS`

I note too that we use `CONFIG_MODULE_NAME = GC3-PrgEnv/v3` but I’m no sure if this would account for a slow down.

Grenville

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [6 August 2023 11:41 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/11 "2023-08-06T11:41:48Z")

</div>

Many thanks,

I made the change:  
ROSE\_LAUNCHER\_PREOPTS\_XIOS = --het-group=2 --nodes={{XIOS\_NODES}} --ntasks={{XIOS\_TASKS}} --tasks-per-node=4 --cpus-per-task=32 --cpu-bind=cores --hint=nomultithread --distribution=block:block --export=all,OMP\_NUM\_THREADS=1,HYPERTHREADS=1  
{% endif %}

but still get similar error:  
/work/n02/n02/jgrist02/cylc-run/u-cy520/log/job/19501001T0000Z/coupled/01/job.err

??? WARNING ???  
? Warning code: -1  
? Warning from routine: ASAD\_CINIT  
? Warning message: NRSTEPS IS OUT OF RANGE, RESETTING  
? Warning from processor: 0  
? Warning number: 68  
???

slurmstepd: error: \*\*\* STEP 4129185.0+0 ON nid003176 CANCELLED AT 2023-08-04T18:40:16 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* JOB 4129185 ON nid003176 CANCELLED AT 2023-08-04T18:40:16 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* STEP 4129185.0+1 ON nid003192 CANCELLED AT 2023-08-04T18:40:16 DUE TO TIME LIMIT \*\*\*  
slurmstepd: error: \*\*\* STEP 4129185.0+2 ON nid003240 CANCELLED AT 2023-08-04T18:40:16 DUE TO TIME LIMIT \*\*\*  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.  
srun: error: nid003243: tasks 14-15: Terminated  
srun: launch/slurm: \_step\_signal: Terminating StepId=4129185.0+0

I will try changing `CONFIG_MODULE_NAME = GC3-PrgEnv/v3` in suite conf/ machine options to see if helps,

Jeremy

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [7 August 2023 08:59 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/12 "2023-08-07T08:59:09Z")

</div>

Hi  
I tried changing `CONFIG_MODULE_NAME = GC3-PrgEnv/v3` in suite conf/ machine options  
but still getting similar time limit error:

from /work/n02/n02/jgrist02/cylc-run/u-cy520/log/job/19501001T0000Z/coupled/01/job.err  
? Warning code: -1

? Warning from routine: ASAD\_CINIT

? Warning message: NRSTEPS IS OUT OF RANGE, RESETTING

? Warning from processor: 0

? Warning number: 68

???

slurmstepd: error: \*\*\* STEP 4149716.0+0 ON nid001257 CANCELLED AT 2023-08-06T17:47:00 DUE TO TIME LIMIT \*\*\*

srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

slurmstepd: error: \*\*\* JOB 4149716 ON nid001257 CANCELLED AT 2023-08-06T17:47:00 DUE TO TIME LIMIT \*\*\*

srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

slurmstepd: error: \*\*\* STEP 4149716.0+1 ON nid001284 CANCELLED AT 2023-08-06T17:47:00 DUE TO TIME LIMIT \*\*\*

slurmstepd: error: \*\*\* STEP 4149716.0+2 ON nid001299 CANCELLED AT 2023-08-06T17:47:00 DUE TO TIME LIMIT \*\*\*

srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

srun: error: nid001307: task 15: Terminated

srun: launch/slurm: \_step\_signal: Terminating StepId=4149716.0+0

thanks,  
Jeremy

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [8 August 2023 12:34 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/13 "2023-08-08T12:34:04Z")

</div>

Jeremy

See` /home/grenville/u-cy520/site/archer2.rc` where in `ROSE_LAUNCHER_PREOPTS_XIOS`, there is no `--cpu-bind=cores`, also in `ROSE_LAUNCHER_PREOPTS_NEMO` there is `OMP_PLACES=cores`

My copy of your suite with these changes ran at normal speed.

Grenville

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [9 August 2023 08:40 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/14 "2023-08-09T08:40:15Z")

</div>

Thanks Grenville,  
I made these changes and get slightly different error fail:  
/work/n02/n02/jgrist02/cylc-run/u-cy520/log/job/19501001T0000Z/coupled/01/job.err

snippet from file:

???  
??? WARNING ???  
? Warning code: -1  
? Warning from routine: ASAD\_CINIT  
? Warning message: NRSTEPS IS OUT OF RANGE, RESETTING  
? Warning from processor: 0  
? Warning number: 68  
???

terminate called after throwing an instance of ‘xios::CException’

???  
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!  
???  
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!  
? Error code: 24  
???  
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!  
? Error code: 24  
? Error from routine: WRITHEAD  
? Error message: WRITHEAD: Addressing conflict

? Error code: 24  
? Error from routine: WRITHEAD  
? Error message: WRITHEAD: Addressing conflict  
? Error from processor: 353  
? Error from processor: 967  
? Error number: 68  
???

? Error from routine: WRITHEAD  
? Error message: WRITHEAD: Addressing conflict  
? Error from processor: 80  
? Error number: 68  
???

? Error number: 68  
???

[967] exceptions: An non-exception application exit occured.  
[967] exceptions: whilst in a serial region  
[967] exceptions: Task had pid=30363 on host nid004000  
[967] exceptions: Program is “/mnt/lustre/a2fs-work2/work/n02/n02/jgrist02/cylc-run/u-cy520/work/19501001T0000Z/coupled/./atmos.exe”  
???  
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!  
? Error code: 24  
? Error from routine: WRITHEAD  
? Error message: WRITHEAD: Addressing conflict  
? Error from processor: 478  
? Error number: 68  
???

[80] exceptions: An non-exception application exit occured.  
[80] exceptions: whilst in a serial region  
[80] exceptions: Task had pid=4016 on host nid001680  
[80] exceptions: Program is “/mnt/lustre/a2fs-work2/work/n02/n02/jgrist02/cylc-run/u-cy520/work/19501001T0000Z/coupled/./atmos.exe”

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 10:10 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/15 "2023-08-09T10:10:16Z")

</div>

Hi Jeremy

You are finding errors that we have fixed in CANARI suites - what are you trying to run, maybe we have a suite already configured to do what you need ?

Grenville

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [9 August 2023 10:19 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/16 "2023-08-09T10:19:48Z")

</div>

ultimately - I just want to run an ensemble of 6 month long simulations - (the experiment bit is changes to the ocean initial conditions )-  
As long as it is ORCA025 - N216, I would hope it would be ok.

If you have something like that which runs, that would be great

Jeremy

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [11 August 2023 09:43 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/17 "2023-08-11T09:43:36Z")

</div>

Hi Jeremy

u-cn134 is the latest CANARI suite – however, it is set up to reconfigure and run from a UM 10.6 start file - I believe u-cy520 is set up the same way, but is reconfiguring a UM 11.6 start file. That is the cause of the error you see. The work around is to not reconfigure `/work/n02/n02/jgrist02/cylc-run/reinhard_restarts/cw475a.da19501001_00`.

switch off reconfiguration, and set `astart = /work/n02/n02/jgrist02/cylc-run/reinhard_restarts/cw475a.da19501001_00`

We will investigate why the reconfiguration mishandles 11.6 files.

Grenville

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [11 August 2023 10:48 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/18 "2023-08-11T10:48:08Z")

</div>

hi Grenville,

I tried those 2 things:  
1)suite conf \>Domain Decomposition\> Build and run\> ’ Run Reconfiguration’ switched to false  
2)um\>namelist\>Model Input and Output\>Dumping and Meaning\> ‘astart’ set to /work/n02/n02/jgrist02/cylc-run/reinhard\_restarts/cw475a.da19501001\_00

However, now I get a quicker fail at fcm\_make2\_um:  
/work/n02/n02/jgrist02/cylc-run/u-cy520/log/job/19501001T0000Z/fcm\_make2\_um/01/job.err

[FAIL] Reading from filehandle failed at /mnt/lustre/a2fs-work1/work/y07/shared/umshared/software/fcm-2019.09.0/bin/…/lib/FCM/Util.pm line 257.  
[FAIL] compile+ 0.0 ! HALO\_EXCHANGE\_MPI\_MOD.mod ← um/src/control/mpp/halo\_exchange\_mpi\_mod.F90  
[FAIL] Reading from filehandle failed at /mnt/lustre/a2fs-work1/work/y07/shared/umshared/software/fcm-2019.09.0/bin/…/lib/FCM/Util.pm line 257.  
[FAIL] compile+ 0.0 ! HALO\_EXCHANGE\_DDT\_MOD.mod ← um/src/control/mpp/halo\_exchange\_ddt\_mod.F90  
[FAIL] compile ---- ! halo\_exchange.o ← um/src/control/mpp/halo\_exchange.F90  
[FAIL] ! HALO\_EXCHANGE.mod : depends on failed target: halo\_exchange.o  
[FAIL] ! HALO\_EXCHANGE\_DDT\_MOD.mod: update task failed  
[FAIL] ! HALO\_EXCHANGE\_MPI\_MOD.mod: update task failed  
[FAIL] ! halo\_exchange.o : depends on failed target: HALO\_EXCHANGE\_DDT\_MOD.mod  
[FAIL] ! halo\_exchange.o : depends on failed target: HALO\_EXCHANGE\_MPI\_MOD.mod

[FAIL] fcm make -C /work/n02/n02/jgrist02/cylc-run/u-cy520/share/fcm\_make\_um -n 2 -j 6 # return-code=2  
2023-08-11T10:29:11Z CRITICAL - failed/EXIT

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [11 August 2023 11:28 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/19 "2023-08-11T11:28:19Z")

</div>

Jeremy

This error is not related to anything you have done. You can just retrigger - but there is no need to keep rebuilding the model, switch off the UM, Drivers, and Ocean builds.

Grenville

---

<div class="post-metadata">

**Author:** ![jgrist02](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@jgrist02](https://cms-helpdesk.ncas.ac.uk/u/jgrist02)\
**Post date:** [15 August 2023 09:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084/20 "2023-08-15T09:54:38Z")

</div>

Many thanks -

I have tried that - and also tried it from the start again, with what seems like a similar error.  
/work/n02/n02/jgrist02/cylc-run/u-cy520/log/job/19501001T0000Z/coupled/01/job.err

???  
??? WARNING ???  
? Warning code: -1  
? Warning from routine: ASAD\_CINIT  
? Warning message: NRSTEPS IS OUT OF RANGE, RESETTING  
? Warning from processor: 0  
? Warning number: 68  
???

terminate called after throwing an instance of ‘xios::CException’  
srun: error: nid006753: task 18: Aborted (core dumped)  
srun: launch/slurm: \_step\_signal: Terminating StepId=4237388.0+0  
srun: launch/slurm: \_step\_signal: Terminating StepId=4237388.0+2  
srun: launch/slurm: \_step\_signal: Terminating StepId=4237388.0+1  
slurmstepd: error: \*\*\* STEP 4237388.0+2 ON nid006680 CANCELLED AT 2023-08-14T13:06:15 \*\*\*  
slurmstepd: error: \*\*\* STEP 4237388.0+0 ON nid002421 CANCELLED AT 2023-08-14T13:06:15 \*\*\*  
slurmstepd: error: \*\*\* STEP 4237388.0+1 ON nid005759 CANCELLED AT 2023-08-14T13:06:15 \*\*\*  
srun: error: nid006685: task 15: Terminated  
srun: error: nid006680: task 1: Terminated  
srun: error: nid006753: task 17: Terminated  
srun: error: nid006680: tasks 2-3: Terminated  
srun: error: nid006683: tasks 10-11: Terminated

[Next page](https://cms-helpdesk.ncas.ac.uk/t/cancelled-due-to-time-limit/1084.md?page=2)