Tasks failing due to archer library errors

Unified Model
, ,
1

Hi CMS,

I’ve just noticed that my IVT files have not been transferring to jasmin and the calc_IVT app has been failing (without actually failing!). Looks like a conda environment or library error - this is the output of job.err for calc_IVT:

Traceback (most recent call last):
File “/home/n02/n02/dch/cfdm/cfdm/core/init.py”, line 29, in
import netCDF4
File “/work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/init.py”, line 3, in
from ._netCDF4 import *
ImportError: /work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/…/…/…/libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/n02/n02/dch/cf-python/cf/init.py”, line 95, in
import cfdm
File “/home/n02/n02/dch/cfdm/cfdm/init.py”, line 44, in
from . import core
File “/home/n02/n02/dch/cfdm/cfdm/core/init.py”, line 31, in
raise ImportError(_error0 + str(error1))
ImportError: cfdm.core requires the modules numpy, netCDF4, packaging. /work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/…/…/…/libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/work/n02/n02/shakka/create_IVT_vars.py”, line 2, in
import cf
File “/home/n02/n02/dch/cf-python/cf/init.py”, line 97, in
raise ImportError(_error0 + str(error1))
ImportError: cf v3.14.1 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm.core requires the modules numpy, netCDF4, packaging. /work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/…/…/…/libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
~
~

Did archer update the software environment recently in a way that would affect these libraries and affect this h5 reference? I’m able to activate the conda environment from the command line without errors, and also run the create_IVT_vars.py script which the calc_IVT app calls interactively, however when I try to call calc_IVT.sh I get errors about cdo not being installed. I can only run it if I ‘module load cdo’ beforehand, which I never had to do before because I had an install of cdo in my conda environment. However, cdo isn’t referenced in the error message above so I’m a bit confused about what’s going on. Can you help? I desperately need this data in January so I’m getting a bit worried about making sure the suite is running properly.

And relatedly, how can I change the calc_IVT app itself so that it fails properly if it is is unable to create the files I need? To save myself the stress in future I’d like to get a nice red fail icon so the suite can’t progress unless it produces the right files.

Thanks,
Ella

2

Hi Ella

There is a mix of python environments going on - the suite sets

[[calc_IVT]]
    inherit = HPC_SERIAL
    pre-script = """
            export PATH=/home/n02/n02/dch/cf/bin:$PATH
            export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH
            """

but then in main.sh

conda activate /work/n02/n02/shakka/IVT_env

Why is this needed - there’s nothing in create_IVT_vars.py other than standard stuff and cf?

Grenville

3

Originally this was needed because it conflicted with my get_era5 environment, I think this is what you recommended here…?

4

The problem in the other ticket was the same kind - the suite was mixing up python environments, so I suggested (indirectly) removing the init-script from HPC_SERIAL.

The latest problem comes from mixing

export PATH=/home/n02/n02/dch/cf/bin:$PATH
export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH

with the IVT_env conda environment. I don’t see why main.sh needs to activate it. (I may be missing the boat of course)

Grenville

5

Hi Grenville,

Thanks, I tried removing the path reference from calc_IVT in suite.rc and still got the same error, which makes me wonder if it is actually the install of netcdf or cf-python.

Traceback (most recent call last):
File “/work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/cfdm/core/init.py”, line 29, in
import netCDF4
File “/work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/init.py”, line 3, in
from ._netCDF4 import *
ImportError: /work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/…/…/…/libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/cf/init.py”, line 95, in
import cfdm
File “/work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/cfdm/init.py”, line 44, in
from . import core
File “/work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/cfdm/core/init.py”, line 31, in
raise ImportError(_error0 + str(error1))
ImportError: cfdm.core requires the modules numpy, netCDF4, packaging. /work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/…/…/…/libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/work/n02/n02/shakka/create_IVT_vars.py”, line 2, in
import cf
File “/work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/cf/init.py”, line 97, in
raise ImportError(_error0 + str(error1))
ImportError: cf v3.15.4 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm.core requires the modules numpy, netCDF4, packaging. /work/n02/n02/shakka/IVT_env/lib/python3.9/site-packages/netCDF4/…/…/…/libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory

6

Have you got cf-python in the IVT_env ?

7

I do, which is why getting rid of the reference to the shared cf python should work…

8

when i have activated just /IVT_env and module load cdo, it runs from the command line. So I guess the other version of cf-python is confusing everything (I’ve forgotten why it was in there)? I’ll try another suite reload…

9

Another update: I tried removing the call to the IVT_env and using only the cf-python install linked under HPC_SERIAL and get the following:

Traceback (most recent call last):
File “/work/n02/n02/shakka/create_IVT_vars.py”, line 2, in
import cf
File “/home/n02/n02/dch/cf-python/cf/init.py”, line 92
_error0 = f"cf v{ version} requires the modules {x}. "
^

10

And if I try with /IVT_env only, I get this netcdf-related issue again.

11

pl tell us which suite?

12

suite is u-cy223. It looks from closer inspection like it could be a conda environment issue? I’m now getting an error about package inconsistencies…

p.s. help, very confused

13

UPDATE: I managed to get it to run by starting from scratch and re-creating the conda environment to resolve the conflicts between the install of netcdf4 and cf-python.

For reference, the environment is /work/n02/n02/shakka/IVT_env2

14

Excellent! glad you got it going. Python is a bit painful on occasion.

Grenville

15

Jheez, tell me about it. Bane of my life. Am still confused about why it suddenly stopped working… and paranoid it’ll happen again. But I’ll cross my fingers and hope for a christmas miracle! Have a lovely festive break Grenville. E

16

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.