Install of convsh on archer2?

General
1

Hi CMS,

Am currently writing an app to calculate integrated vapour transport online and need first to extract fields from pp and save to netcdf. I was hoping to do this with convsh, but i couldn’t find out if there is already an install on archer2 - if so, can you point me towards it please?

Cheers
Ella

2

Ella

Why not use cf-python?

Grenville

3

Ooh, didn’t know I could… is there a guide to using it anywhere? I need to stashsplit my data into individual .nc files, basically.

4

on ARCHER2:

export PATH=/home/n02/n02/dch/cf/bin:$PATH
export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH

then have a look at:

https://ncas-cms.github.io/cf-python/tutorial.html
https://ncas-cms.github.io/cf-python/tutorial.html#sorting-and-selecting-from-field-lists
https://ncas-cms.github.io/cf-python/tutorial.html#pp-and-um-fields-files

(convsh is on ARCHER2 in /work/y07/shared/umshared/bin)

Grenville

1 Like
5

Thanks for this Grenville. Am using cf in a new app (calc_IVT) which creates .nc files from UM model level variables (u,v,q,P) in pp files, then calculates integrated water vapour transport online using CDO, and then delete the full profiles that I don’t need before I archive to jasmin.

I think I’ve set up all the scripts I need to do this okay, but I’m getting python errors about missing packages like this:

Traceback (most recent call last):
  File "/home/n02/n02/dch/cfdm/cfdm/core/__init__.py", line 29, in <module>
    import netCDF4
  File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ModuleNotFoundError: No module named 'netCDF4._netCDF4'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/n02/n02/dch/cf-python/cf/__init__.py", line 95, in <module>
    import cfdm
  File "/home/n02/n02/dch/cfdm/cfdm/__init__.py", line 44, in <module>
    from . import core
  File "/home/n02/n02/dch/cfdm/cfdm/core/__init__.py", line 31, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/n02/n02/shakka/create_IVT_vars.py", line 2, in <module>
    import cf
  File "/home/n02/n02/dch/cf-python/cf/__init__.py", line 97, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cf v3.14.1 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4'
/work/n02/n02/shakka/cylc-run/u-cz527/app/calc_IVT/bin/main.sh: line 3: cdo: command not found
[FAIL] main.sh <<'__STDIN__'
[FAIL] 
[FAIL] '__STDIN__' # return-code=127
2023-10-07T07:21:55Z CRITICAL - failed/EXIT

App is calc_IVT and suite is u-cz527.

The pre-script in suite.rc is

  pre-script = """
                export PATH=/home/n02/n02/dch/cf/bin:$PATH
                export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH
                """
        env-script = """
                eval $(rose task-env)
                module load cdo
                """

anything else I should add in there to get netCDF4 to work?

6

Ella

please add module load netcdf4

Grenville

7

Tried adding ‘module load netcdf4’ to the env-script, pre-script and main.sh but still doesn’t seem to want to work. I also get the same error if I try to run create_IVT_vars.py directly from the archer command line. Is the cf-python install overwriting the call to load netcdf4 or something?


Traceback (most recent call last):
  File "/home/n02/n02/dch/cfdm/cfdm/__init__.py", line 68, in <module>
    import netcdf_flattener
ModuleNotFoundError: No module named 'netcdf_flattener'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/n02/n02/dch/cf-python/cf/__init__.py", line 95, in <module>
    import cfdm
  File "/home/n02/n02/dch/cfdm/cfdm/__init__.py", line 70, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cfdm requires the modules cftime, netcdf_flattener. No module named 'netcdf_flattener'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/n02/n02/shakka/create_IVT_vars.py", line 2, in <module>
    import cf
  File "/home/n02/n02/dch/cf-python/cf/__init__.py", line 97, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cf v3.14.1 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm requires the modules cftime, netcdf_flattener. No module named 'netcdf_flattener'

cdo (Abort): e[4m/work/n02/n02/shakka/calc_IVT.she[0m -c 19991231T1200Z -s u-cz527  
                             ^ Operator not found
[FAIL] main.sh <<'__STDIN__'
[FAIL] 
[FAIL] '__STDIN__' # return-code=1
2023-10-08T13:41:55Z CRITICAL - failed/EXIT
8

pl give us read permission on /work/n02/n02/shakka/create_IVT_vars.py

9

Done. Have also give permissions on calc_IVT.sh.

10

Ella - this looks like an environment problem – remove from your .bashrc
export PATH="/home/n02/n02/shakka/.local/bin:$PATH"

and possibly
export PATH="/work/n02/n02/shakka/:$PATH"

there is no need to module load netcdf4 (that was a mistake on my part)

Grenville

11

I think those lines are the ones you suggested I add to my .bashrc to allow the get_era_data app (esp cdsapi) to work - will removing this interfere with that?

12

It looks like you’ll need separate environments for the the various tasks …

13

That’s sensible. Is the preferred method on archer2 to use conda environments?

14

Hi Grenville,

Have created separate conda environments to use cf-python and cdsapi now. But i am now getting an error about a rose-app task call to python2 in the /y07/shared directory:

/work/y07/shared/umshared/software/rose-pumatest/bin/rose-task-run: line 57: exec: python2: not found
/usr/bin/env: ‘python2’: No such file or directory

I can’t see where this is coming from exactly, and I’ve removed all the pre-script and env-script lines except the calls to activate the relevant conda environment and then module load cdo. I’m guessing there’s some rose script that’s still running python 2.7 somewhere?

Any ideas how to resolve this?

Cheers
Ella

15

Ella

Please chmod -R g+rX /home/n02/n02/shakka on puma2

Grenville

16

what is the suite id?

17

Suite id is u-cz527, have granted permissions.

FYI the two environments work interactively (/work/n02/n02/shakka/ERA5_env and …/IVT_env) but I’m just struggling to figure out how best to call them in the apps.

18

Am getting this now:

CommandNotFoundError: Your shell has not been properly configured to use ‘conda activate’.
To initialize your shell, run

$ conda init <SHELL_NAME>

Currently supported shells are:

  • bash
  • fish
  • tcsh
  • xonsh
  • zsh
  • powershell

See ‘conda init --help’ for more information and options.

IMPORTANT: You may need to close and restart your shell after running ‘conda init’.

19

Update: got it to run using #!/bin/bash --login in my .bashrc
but now the script is not behaving as expected, so digging into that now…

20

Ella

[grenvill@puma2 ~]$ cd ~shakka
-bash: cd: /home/n02/n02/shakka: Permission denied