# Python errors in UKESM1.2

**URL:** <https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836>\
**Category:** Uncategorized\
**Tags:** ARCHER2\
**Created:** [20 October 2025 08:56 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836 "2025-10-20T08:56:50Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [20 October 2025 08:56 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/1 "2025-10-20T08:56:50Z")

</div>

Hi CMS team,

My suites u-dr800 and u-dr928 both fail with some python errors that didn’t use to occur.. have there been any python updates on ARCHER2?

the tasks that fail are

unicicles\_gris; unicicles\_ais; postproc\_atmos

For the unicicles tasks, the errors are:

`srun: error: nid005710: tasks 0-21,23-47,49-127: Exited with exit code 127 srun: launch/slurm: _step_signal: Terminating StepId=11209862.0 /work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_unicicles/unicicles/bin/unicicles: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory /work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_unicicles/unicicles/bin/unicicles: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory slurmstepd: error: ***STEP 11209862.0 ON nid005710 CANCELLED AT 2025-10-16T18:01:37*** srun: error: nid005710: tasks 22,48: Exited with exit code 127 [FAIL] $IEXECDIR/unicicles_wrapper # return-code=127 2025-10-16T17:01:38Z CRITICAL - failed/ERR`

Note this happened before, this was a fix from Grenville:

in `site/archer2-unicicles.rc` add `--export=all `thus  
`ROSE_LAUNCHER_PREOPTS = --hint=nomultithread --distribution=block:block --cpu-bind=cores --nodes=1 --ntasks=128 --export=all`in `[[UNI_EXEC_RESOURCE]]`

This currently is included in the suite.  
Then Grenville wrote:

~~same for [[CAP\_RESOURCE]] in site/archer2-unicicles.rc~~  
~~ROSE\_LAUNCHER\_PREOPTS = --hint=nomultithread --distribution=block:block --cpu-bind=cores --nodes=1 --ntasks=1 **–export=all**~~ Fixed elsewhere - nothing needed

Could it be that this now needs an update elsewhere? I saw another ticket about issues with CAP ( [Issue with CAP9.1 after archer2 update](https://cms-helpdesk.ncas.ac.uk/t/issue-with-cap9-1-after-archer2-update/1834) )

For postproc\_atmos, the error I get is:

`Traceback (most recent call last):`  
`File “/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/main_pp.py”, line 119, in `  
`main()`  
`File “/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/main_pp.py”, line 112, in main`  
`run_postproc()`  
`File “/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/main_pp.py”, line 83, in run_postproc`  
`getattr(model, meth)()`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/timer.py”, line 115, in wrapper`  
`out = function(*args, **kw)`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/atmos.py”, line 491, in do_meaning`  
`icode = self.update_meanfile(meanfile, setend)`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/timer.py”, line 115, in wrapper`  
`out = function(*args, **kw)`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/atmos.py”, line 445, in update_meanfile`  
`rcode = climatemean.create_mean(meanfile,`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/timer.py”, line 115, in wrapper`  
`out = function(*args, **kw)`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/climatemean.py”, line 256, in create_mean`  
`icode, output = target_app(meanfile, **kwargs)`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/timer.py”, line 115, in wrapper`  
`out = function(*args, **kw)`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/atmos_transform.py”, line 275, in create_um_mean`  
`load_mule = [mule.load_umfile(f) for f in meanfile.component_files]`  
`File “/mnt/lustre/a2fs-work2/work/n02/n02/adittus/cylc-run/u-dr928/run2/share/fcm_make_pp/build/bin/atmos_transform.py”, line 275, in `  
`load_mule = [mule.load_umfile(f) for f in meanfile.component_files]`  
`File “/work/y07/shared/umshared/lib/python3.9/mule/` **`init`** `.py”, line 1845, in load_umfile`  
`result = _load_umfile(file_path, open_file)`  
`File “/work/y07/shared/umshared/lib/python3.9/mule/` **`init`** `.py”, line 1836, in _load_umfile`  
`raise ValueError(msg)`  
`ValueError: Unknown dataset_type 0, supported types are dict_keys([1, 2, 3, 4, 5])`  
`[FAIL] main_pp.py atmos # return-code=1`  
`2025-10-17T21:20:43Z CRITICAL - failed/ERR`

Thanks for any help!

Cheers,  
Andrea

---

<div class="post-metadata">

**Author:** ![Robin\_Smith](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/robin_smith/32/75_2.png) [@Robin\_Smith](https://cms-helpdesk.ncas.ac.uk/u/Robin_Smith)\
**Post date:** [20 October 2025 09:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/2 "2025-10-20T09:15:25Z")

</div>

For the unicicles ones I modified the “module load cray-python” in my site/archer2-unicicles.rc to include the version number, ie “module load cray-python/3.9.13.1”, but that won’t help the postproc

---

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [20 October 2025 10:00 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/3 "2025-10-20T10:00:46Z")

</div>

Thanks Robin, re-running the unicicles tasks now!

---

<div class="post-metadata">

**Author:** ![AnnetteOsprey](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/annetteosprey/32/27_2.png) [@AnnetteOsprey](https://cms-helpdesk.ncas.ac.uk/u/AnnetteOsprey)\
**Post date:** [20 October 2025 10:23 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/4 "2025-10-20T10:23:26Z")

</div>

Hi Andrea,

If you look in the `job.out` for the postproc\_atmos task you can see what it was trying to do when it failed:

```auto
[INFO] Creating meanfile dr800a.py20711201 with components:
        /work/n02/n02/adittus/cylc-run/u-dr800/run2/share/data/History_Data/dr800a.ps2071djf
        /work/n02/n02/adittus/cylc-run/u-dr800/run2/share/data/History_Data/dr800a.ps2071jja
        /work/n02/n02/adittus/cylc-run/u-dr800/run2/share/data/History_Data/dr800a.ps2071mam
        /work/n02/n02/adittus/cylc-run/u-dr800/run2/share/data/History_Data/dr800a.ps2071son

```

The mule error `ValueError: Unknown dataset_type 0` suggests that it can’t read a file, and indeed if we look at those seasonal means, the `djf` file hasn’t been written properly.

```auto
archer2 History_Data$ ls -l dr800a.ps* 
-rw-r--r-- 1 adittus n02 563986432 Oct 16 17:28 dr800a.ps2071djf
-rw-r--r-- 1 adittus n02 5139582976 Oct 16 21:55 dr800a.ps2071jja
-rw-r--r-- 1 adittus n02 5139464192 Oct 16 19:39 dr800a.ps2071mam
-rw-r--r-- 1 adittus n02 5139722240 Oct 17 00:10 dr800a.ps2071son

```

Annette

---

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [21 October 2025 08:51 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/5 "2025-10-21T08:51:56Z")

</div>

Thank you Annette. Now this task is complaining because it can’t make the annual mean without the djf file, and I suspect it was a previous task that generated the djf file. Is there a way I can tell the suite to skip the seasonal means and annual means for this cycle?

If I set the task to succeeded manually, will it fail to do any other important tasks?

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [21 October 2025 09:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/6 "2025-10-21T09:55:23Z")

</div>

Andrea

If you don’t want seasonal and annual means at all (I though that’s what you wanted), try switching them off in postproc→Atmosphere→File transformation

(set create\_means to false), reload the suite, retrigger the task.

Setting the task to succeeded may not work - hard to know if it has completed everything else it’s supposed to have done.

Grenville

---

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [21 October 2025 10:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/7 "2025-10-21T10:58:28Z")

</div>

Thanks Grenville, have done that now.  
Now I get:

`[WARN] Iris Module is not available`  
`[ERROR] Validity time mismatch in file /work/n02/n02/adittus/cylc-run/u-dr800/run2/share/data/History_Data/dr800a.pm2071nov to be archived`  
` → Expected [2071, 11, 1, 0, 20, 0] and got [2071, 12, 1]`  
`[FAIL] Command Terminated`

Any ideas how to fix this? If I re-run the coupled task, is that more likely to fix things or make them worse?

Thanks,  
Andrea

---

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [4 November 2025 15:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/8 "2025-11-04T15:58:54Z")

</div>

Hi CMS,

Thanks for the replies on this thread. After talking with Grenville and Robin, it seems the easiest for u-dr800 would be to restart it in January of the failure, as the problems in postproc can’t easily be resolved. As the suite has run a long way ahead, I have copied the suite, and wanted to re-run just the year in question (new suite-ID u-dt871). I’d then take the re-generated files (corrupted in u-dr800), rename and move to u-dr800 to just re-run the postproc task.

The trouble I’m having now is that the files in u-dt871 do not bit-compare with u-dr800 pre-failure, so this plan failed.

Is there anything I’m missing?

NRUN\_BITCOMP=TRUE  
RECON=False

BASIS=2071,1,1,0,0,0

Updated values for:

atmos\_dr800c\_P1Y\_20700101-20710101\_icecouple.nc - DONE  
bisicles\_dr800c\_20710101\_bathymetry-isf.nc - DONE  
bisicles\_dr800c\_20710101\_restart-AIS.hdf5 - DONE  
bisicles\_dr800c\_20710101\_restart-GrIS.hdf5 - DONE  
bisicles\_dr800c\_P1Y\_20700101-20710101\_calving-AIS.hdf5 - DONE  
bisicles\_dr800c\_P1Y\_20700101-20710101\_calving-GrIS.hdf5 - DONE  
bisicles\_dr800c\_P1Y\_20700101-20710101\_calving.nc - DONE

dr800a.da20710101\_00 - DONE (this is an environment variable in suite, so I renamed and moved it to /work/n02/n02/adittus/cylc-run/u-dt871/run1/share/data/dt871.astart)

dr800i.restart.2071-01-01-00000.nc - DONE  
dr800o\_20710101\_restart.nc -DONE  
dr800o\_20710101\_restart\_trc.nc - DONE  
dr800o\_icebergs\_20710101\_restart.nc - DONE  
glint\_dr800c\_20710101\_restart-AIS.nc - DONE  
glint\_dr800c\_20710101\_restart-GrIS.nc - DONE  
nemo\_dr800c\_P1Y\_20700101-20710101\_icecouple.nc - DONE

Any pointers would be greatly appreciated!

Thanks,  
Andrea

---

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [4 November 2025 15:59 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/9 "2025-11-04T15:59:32Z")

</div>

I did recompile the executables.

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [5 November 2025 10:04 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/10 "2025-11-05T10:04:28Z")

</div>

Andrea

In rose-app-conf, u-dt871 has stphseed=0, which is odd (it should be 2) - it looks like the bit compare override has not been applied.

in flow.cylc, in [[COUPLED]], set

ROSE\_APP\_OPT\_CONF\_KEYS = {{RM\_MEDUSA\_OPT\_KEY}} {{UNICICLES\_OPT\_KEY}} {{BITCOMP\_NRUN\_OPT}}

I think that will work (set stphseed at least)

Grenville

---

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [6 November 2025 10:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/11 "2025-11-06T10:54:30Z")

</div>

Thanks Grenville, I will try this.  
Right now all my tasks have a submit-fail error with no error message.. Is there anything going on that could be causing this, or is this likely to be on my end (note the suite ran before, so unlikely to be my configuration)

Thanks,  
Andrea

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [6 November 2025 11:48 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/12 "2025-11-06T11:48:30Z")

</div>

on puma, can you

ssh ln0[1,2,3,4] ?

---

<div class="post-metadata">

**Author:** ![andreadittus](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/andreadittus/32/461_2.png) [@andreadittus](https://cms-helpdesk.ncas.ac.uk/u/andreadittus)\
**Post date:** [6 November 2025 12:03 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/13 "2025-11-06T12:03:14Z")

</div>

> [@grenville](#):
>
> ssh ln0[1

I can for all but ln01, which gives me a “Connection closed by ….” error message

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [6 November 2025 12:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/14 "2025-11-06T12:15:01Z")

</div>

You’re out of space on /work

---

<div class="post-metadata">

**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/1fd2411499ffcbc299fe756cd5cdf26e44956558.png) [@system](https://cms-helpdesk.ncas.ac.uk/u/system)\
**Post date:** [6 December 2025 12:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/python-errors-in-ukesm1-2/1836/15 "2025-12-06T12:15:57Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.