# UMESM1.1 failing **URL:** \ **Category:** Unified Model\ **Created:** [2 November 2023 17:20 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238 "2023-11-02T17:20:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [2 November 2023 17:20 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/1 "2023-11-02T17:20:12Z")
Hi , I have made a copy of u-cj514/trunk@264964 to give u-da865. I have made the recommended changes at [UKESM1.1-AMIP Release Notes](https://cms.ncas.ac.uk/unified-model/configurations/ukesm/relnotes-1.1/amip/). The model crashes soon after starting with the error below. Are there other things I need to change to make it work? Simon ??? ???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!! ? Error code: 2 ? Error from routine: EG\_BICGSTAB\_MIXED\_PREC ? Error message: Convergence failure in BiCGstab after restartSee the following URL for more information: ? [https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints](https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints) ? Error from processor: 377 ? Error number: 61 ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [3 November 2023 08:40 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/2 "2023-11-03T08:40:54Z")
Hi Simon, There a couple of things that need changing in that suite. I’m currently talking to the MO to get that suite (and others) updated. You’ll need to change the suite to pick up the start dump from Simon’s directory for now. `/work/n02/n02/simon/u-by791/by791a.da19790101_00` In file `site/archer2.rc` you’ll also need to change: `ROSE_LAUNCHER_PREOPTS = {{ATM_SLURM_FLAGS}}` to `ROSE_LAUNCHER_PREOPTS = {{ATM_SLURM_FLAGS}} --cpus-per-task={{MAIN_OMPTHR_ATM}}` Cheers, Ros. ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [3 November 2023 10:45 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/3 "2023-11-03T10:45:35Z")
Thanks Ros, I’ll do that and report back! The failure message doesn’t look like a file-not-found error – is it falling back to some unreasonable default? Simon ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [3 November 2023 14:38 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/4 "2023-11-03T14:38:43Z")
And it seems to be working – model is running and ran for a while! Simon ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [3 November 2023 14:59 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/5 "2023-11-03T14:59:59Z")
Glad to hear it’s running now. It wasn’t a file-not-found issue, the start dump was dodgey. ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [6 November 2023 11:11 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/6 "2023-11-06T11:11:55Z")
Though the model was running it ran out of time after 3 hours & 50 mins. Which I think should be enough for a 3 month simulation. Looking in the output the model has ran 311 timesteps and the pa/pm/pb files are empty according to xconv. job.err has messages of the form: srun: Warning: can’t honor --ntasks-per-node set to 64 which doesn’t match the requested tasks 504 with the number of requested nodes 8. Ignoring --ntasks-per-node. [0] exceptions: feenableexcept() mask [0x00000000] enabled. (mask [0x00000000] requested) WARNING: Requested total thread count and/or thread affinity may result in oversubscription of available CPU resources! Performance may be degraded. Any advice much appreciated. Simon ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [6 November 2023 12:40 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/7 "2023-11-06T12:40:50Z")
Hi Simon In site/archer2.ac, add the --cpus-per-task clause like this: ```auto {% if MAIN_OMPTHR_ATM > 1 %} {% set ATM_SLURM_FLAGS= "--hint=nomultithread --distribution=block:block --cpus-per-task={{MAIN_OMPTHR_ATM}}" %} {% else %} {% set ATM_SLURM_FLAGS = "--cpu-bind=cores" %} {% endif %} ``` Grenville ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [7 November 2023 10:11 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/8 "2023-11-07T10:11:51Z")
Hi Grenville, that failed. Job hang around for ages and then looks to have failed with the following error in job.err: srun: error: Invalid numeric value “{{MAIN\_OMPTHR\_ATM}}” for --cpus-per-task. [FAIL] um-atmos \<\<‘ **STDIN** ’ [FAIL] [FAIL] ‘ **STDIN** ’ # return-code=1 2023-11-07T01:09:20Z CRITICAL - failed/EXIT I don’t understand rosie so can’t tell if I made a small error… Simon ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [7 November 2023 14:41 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/9 "2023-11-07T14:41:34Z")
Simon _Mea culpa_ - remove the ` --cpus-per-task={{MAIN_OMPTHR_ATM}}` from `ATM_SLURM_FLAGS` and add it to `ROSE_LAUNCHER_PREOPTS = {{ATM_SLURM_FLAGS}}` in `ATMOS_RESOURSE` section. My jinja knowledge is weaker than I’d thought. Grenville ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [8 November 2023 09:29 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/10 "2023-11-08T09:29:03Z")
> [@grenville](#): > > `--cpus-per-task={{MAIN_OMPTHR_ATM}}` So becomes: ROSE\_LAUNCHER\_PREOPTS = {{ATM\_SLURM\_FLAGS}} --cpus-per-task={{MAIN\_OMPTHR\_ATM}} ?? ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [8 November 2023 09:42 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/11 "2023-11-08T09:42:36Z")
yes - (like you have for the reconfiguration) ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [8 November 2023 16:50 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/12 "2023-11-08T16:50:50Z")
And that has worked. Guess I must have been the first person to use this configuration in a while… As I want to build on this, should I push the whole thing back into fcm? Simon ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [8 November 2023 17:32 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/13 "2023-11-08T17:32:13Z")
OK – I don’t understand fcm so when I copy my modified job the modifications do not propagate… I have done fcm ci on the job. ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [9 November 2023 07:22 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/14 "2023-11-09T07:22:07Z")
Hi Simon, The suite has been copied from the wrong version u-da865@r270554 - you checked in your changes at r271439. On the command line do `rosie copy u-da865` this will copy from the head of the suite. Cheers, Ros ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [9 November 2023 15:30 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/15 "2023-11-09T15:30:58Z")
Thanks a lot Ros. Though as usual I had to kick off my met office code password… Simon ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [9 November 2023 16:46 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/16 "2023-11-09T16:46:37Z")
Hi Simon, Yes, the MOSRS password is only cached for 12hours before it requires you to re-cache it. Cheers, Ros. ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [9 November 2023 16:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/17 "2023-11-09T16:58:25Z")
That is so annoying. Why can’t they use ssh keys etc… ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [10 November 2023 09:19 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/18 "2023-11-10T09:19:47Z")
I’ve moved to case u-db167 which is a version with no PP output and netcdf output. Case runs and produces output. But at the end I get an error: /work/n02/n02/tetts/cylc-run/u-db167/bin/save\_wallclock.sh: /work/n02/n02/tetts/cylc-run/u-db167/bin/iteration\_bins.py: /usr/bin/python: bad interpreter: No such file or directory I think this is because /work/n02/n02/tetts/cylc-run/u-db167/bin/iteration\_bins.py uses a full path to python as opposed to #!/usr/bin/env python. tetts@ln01:/work/n02/n02/tetts/cylc-run/u-db167\> type -p python /work/y07/shared/utils/core/python/miniconda2/bin/python My python is somewhere else. Anyhow, can the interation\_bins.py script be modified to use #!/usr/bin/env python ta Simon ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [10 November 2023 09:51 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/19 "2023-11-10T09:51:02Z")
Hi SImon Do you need to save the wallclock times ? If not, just remove `post-script = save_wallclock.sh {{EXPT_RESUB}}` Grenville ---
**Author:** ![SimonTett](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@SimonTett](https://cms-helpdesk.ncas.ac.uk/u/SimonTett)\ **Post date:** [10 November 2023 11:19 UTC](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238/20 "2023-11-10T11:19:49Z")
I’ve modified my copy of iteration\_bins.py Simon [Next page](https://cms-helpdesk.ncas.ac.uk/t/umesm1-1-failing/1238.md?page=2)