Mule on ARCHER2 not working with `rose-um-env-puma2` environment

Hello

I use mule quite a bit in my ARCHER2 UM workflow but I’m struggling to get it to work properly at the moment - whenever I try to import it I get

>>> import mule

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "<stdin>", line 1, in <module>
  File "/work/y07/shared/umshared/lib/python3.9/mule/__init__.py", line 1788, in <module>
    from mule.ff import FieldsFile  # noqa: E402
  File "/work/y07/shared/umshared/lib/python3.9/mule/ff.py", line 29, in <module>
    from mule.packing import wgdos_pack_field, wgdos_unpack_field
  File "/work/y07/shared/umshared/lib/python3.9/mule/packing.py", line 46, in <module>
    import um_packing
  File "/work/y07/shared/umshared/lib/python3.9/um_packing/__init__.py", line 21, in <module>
    from .um_packing import wgdos_pack, wgdos_unpack, get_shumlib_version
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
  File "/work/y07/shared/umshared/lib/python3.9/mule/packing.py", line 46, in <module>
    import um_packing
  File "/work/y07/shared/umshared/lib/python3.9/um_packing/__init__.py", line 21, in <module>
    from .um_packing import wgdos_pack, wgdos_unpack, get_shumlib_version
ImportError: numpy.core.multiarray failed to import

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/work/y07/shared/umshared/lib/python3.9/mule/__init__.py", line 1788, in <module>
    from mule.ff import FieldsFile  # noqa: E402
  File "/work/y07/shared/umshared/lib/python3.9/mule/ff.py", line 29, in <module>
    from mule.packing import wgdos_pack_field, wgdos_unpack_field
  File "/work/y07/shared/umshared/lib/python3.9/mule/packing.py", line 107, in <module>
    raise ImportError(err.args + (msg,))
ImportError: ('numpy.core.multiarray failed to import', 'SHUMlib Packing library found, but failed to import')

I already have . /work/y07/shared/umshared/bin/rose-um-env-puma2 in my ~/.bash_profile as suggested in this thread and this has worked fine previously when running interactively and as part of cylc workflows and as part of normal batch scripts.

I understand there seems to be some sort of issue with module clashes with numpy 2, and I can keep trying to find a workaround, but I figured I may as well post here to let you know as this might be an issue with the central environment?

Have a good weekend all!

Fran

NB edit was because the title wasn’t very descriptive; I suppose I should add that I also tried loading the um and postproc modules on ARCHER2 to no avail to get mule to work.

Hi Fran,

The traceback above is when you import on the ARCHER2 command line yes?

I’m struggling to recreate it:
Interactively mule imports fine for me with my .bash_profile or your ~/.bash_profile

ARCHER2-23cab> module load cray-python
ARCHER2-23cab> python
Python 3.9.13 (main, Aug 10 2022, 17:20:06)
[GCC 9.3.0 20200312 (Cray Inc.)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mule
>>>

What suite are you trying to run it in?

Regards,
Ros.

Hi Ros

Thanks for getting back to me!

This error happens when I type the exact same commands as you listed above in either a login node or using an interactive srun session.

An example of it not working on puma2 cylc job submission is when I have recently run suite u-dm226, the transplant step fails with the following error:

Traceback (most recent call last):
  File "/work/n02/n02/franmorr/cylc-run/u-dm226/app/transplant/bin/replace_data_maintain_headers.py", line 195, in <module>
    dump = mule.DumpFile.from_file(dump_file)
  File "/work/y07/shared/umshared/lib/python3.9/mule/__init__.py", line 1254, in from_file
    new_umf._read_file(file_or_filepath)
  File "/work/y07/shared/umshared/lib/python3.9/mule/ff.py", line 502, in _read_file
    super(FieldsFile, self)._read_file(file_or_filepath)
  File "/work/y07/shared/umshared/lib/python3.9/mule/__init__.py", line 1432, in _read_file
    FixedLengthHeader.from_file(source))
  File "/work/y07/shared/umshared/lib/python3.9/mule/__init__.py", line 555, in from_file
    return super(FixedLengthHeader, cls).from_file(source, cls._NUM_WORDS)
  File "/work/y07/shared/umshared/lib/python3.9/mule/__init__.py", line 393, in from_file
    return cls(values)
  File "/work/y07/shared/umshared/lib/python3.9/mule/__init__.py", line 527, in __init__
    raise ValueError(_msg)
ValueError: Incorrect size for fixed length header; given 0 words but should be 256.
[FAIL] python ${CYLC_SUITE_DEF_PATH}/app/transplant/bin/replace_data_maintain_headers.py $ASTART_UNMODIFIED $LAKE_STATE_FILE $ASTART_UNMODIFIED $ASTART_FILENAME $ASTART_TRANSPLANT_FILENAME <<'__STDIN__'
[FAIL] 
[FAIL] '__STDIN__' # return-code=1
2025-06-27T14:05:50Z CRITICAL - failed/EXIT

This could obviously be something else though. What I was trying to do currently was take some lake files out of a start dump so that they can eventually be transplanted (using the above task in a different suite). This was using a combination of python with mule setting up a singularity container to use ANTS, using /work/n02/n02/franmorr/firstrains-um-utils/mule-transplanting/lake-ancil-regrid/regrid_lake_ancils.slurm which leads to different errors involving scipy?!? (see the slurm.out files in the same directory for more details)

I don’t know why this has gone weird, I haven’t changed any settings as far as I’m aware?

Fran

Hi Fran,

The error from the cylc suite is a completely different problem.

"ValueError: Incorrect size for fixed length header; given 0 words but should be 256." is caused by a zero length or corrupt data file. Please check for zero length files.

Regards,
Ros.

Hi Ros

Thanks, yes I did suspect this, and I am trying to set up another transplanting step with mule to test whether it works when submitted as part of a cylc suite, but it doesn’t change the fact that I still can’t use mule from ARCHER2, which is stopping me from the doing the task I need to do (regridding some fields from a start dump using ANTS into netcdf files I can use to insert into a suite).

I don’t want to have to set up a cylc suite to run one regridding task which doesn’t need rerunning systematically.

I’m not sure if I’ve accidentally set up a rogue environment, but I am importing cray-python, and it says Python 3.9.13 (main, Aug 10 2022, 17:20:06) when I open an interactive python window too. When you do a pip list for your cray-python environment, does it also have numpy listed at version 2.0.2? I suppose this is the root of the issue, but I don’t understand why you can’t replicate it…

Thanks very much for your help so far! I really appreciate it!

Hi Fran,

No. It lists numpy/1.21.5 for me.

ARCHER2-23cab> module load cray-python

ARCHER2-23cab> python --version
Python 3.9.13

ARCHER2-23cab> module list

Currently Loaded Modules:
  1) craype-x86-rome            4) perftools-base/22.12.0                  7) craype/2.7.19      10) cray-libsci/22.12.1.1  13) epcc-setup-env
  2) libfabric/1.12.1.2.2.0.0   5) xpmem/2.5.2-2.4_3.30__gd0f7936.shasta   8) cray-dsmml/0.2.2   11) PrgEnv-cray/8.3.3      14) load-epcc-module
  3) craype-network-ofi         6) cce/15.0.0                              9) cray-mpich/8.1.23  12) bolt/0.8               15) cray-python/3.9.13.1

 

ARCHER2-23cab> pip list
Package            Version
------------------ --------
atomicwrites       1.4.0
attrs              21.4.0
Cython             0.29.28
dask               2022.2.1
fsspec             2022.3.0
importlib-metadata 0.0.0
iniconfig          1.1.1
locket             0.2.1
more-itertools     8.10.0
mpi4py             3.1.3
nose               1.3.7
numpy              1.21.5
packaging          21.3
pandas             1.4.2
partd              1.2.0
pip                22.0.4
pluggy             1.0.0
py                 1.11.0
pybind11           2.6.2
pyparsing          3.0.4
pytest             0.0.0
python-dateutil    2.8.2
pytz               2021.3
PyYAML             6.0
scipy              1.6.2
setuptools         58.1.0
setuptools-scm     6.0.1
six                1.16.0
toml               0.10.2
toolz              0.11.2
wcwidth            0.2.5
zipp               0.0.0
WARNING: You are using pip version 22.0.4; however, version 25.1.1 is available.
You should consider upgrading via the '/opt/cray/pe/python/3.9.13.1/bin/python -m pip install --upgrade pip' command.

Cheers,
Ros.

Huh, sure. I made a new environment with numpy 1.26.4 (/work/n02/n02/franmorr/venvs/mule) and it behaves normally. I have no idea why my numpy defaults to 2.0.2 under cray-python but oh well, it’s workable now.

Thanks for the guidance Ros!

Hi Fran,

Hmmmm, that’s very weird that you’ve had to create a virtual environment rather that just use cray-python. Glad you’ve found a workaround though.

Cheers,
Ros.

Ah, I figured it out - I had unknowingly done some terrible pip --user installs which had ruined everything. My bad! Have got rid of those now and this has solved quite a few problems…

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.