# Install of convsh on archer2?

**URL:** <https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155>\
**Category:** General\
**Tags:** ARCHER2\
**Created:** [3 October 2023 14:13 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155 "2023-10-03T14:13:31Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [3 October 2023 14:13 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/1 "2023-10-03T14:13:31Z")

</div>

Hi CMS,

Am currently writing an app to calculate integrated vapour transport online and need first to extract fields from pp and save to netcdf. I was hoping to do this with convsh, but i couldn’t find out if there is already an install on archer2 - if so, can you point me towards it please?

Cheers  
Ella

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [3 October 2023 14:26 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/2 "2023-10-03T14:26:09Z")

</div>

Ella

Why not use cf-python?

Grenville

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [3 October 2023 14:32 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/3 "2023-10-03T14:32:23Z")

</div>

Ooh, didn’t know I could… is there a guide to using it anywhere? I need to stashsplit my data into individual .nc files, basically.

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [3 October 2023 15:27 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/4 "2023-10-03T15:27:17Z")

</div>

on ARCHER2:

```auto
export PATH=/home/n02/n02/dch/cf/bin:$PATH
export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH

```

then have a look at:

```auto
https://ncas-cms.github.io/cf-python/tutorial.html
https://ncas-cms.github.io/cf-python/tutorial.html#sorting-and-selecting-from-field-lists
https://ncas-cms.github.io/cf-python/tutorial.html#pp-and-um-fields-files

```

(convsh is on ARCHER2 in /work/y07/shared/umshared/bin)

Grenville

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [7 October 2023 07:52 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/5 "2023-10-07T07:52:09Z")

</div>

Thanks for this Grenville. Am using cf in a new app (calc\_IVT) which creates .nc files from UM model level variables (u,v,q,P) in pp files, then calculates integrated water vapour transport online using CDO, and then delete the full profiles that I don’t need before I archive to jasmin.

I think I’ve set up all the scripts I need to do this okay, but I’m getting python errors about missing packages like this:

```auto
Traceback (most recent call last):
  File "/home/n02/n02/dch/cfdm/cfdm/core/ __init__.py", line 29, in <module>
    import netCDF4
  File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/netCDF4/ __init__.py", line 3, in <module>
    from ._netCDF4 import *
ModuleNotFoundError: No module named 'netCDF4._netCDF4'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/n02/n02/dch/cf-python/cf/ __init__.py", line 95, in <module>
    import cfdm
  File "/home/n02/n02/dch/cfdm/cfdm/ __init__.py", line 44, in <module>
    from . import core
  File "/home/n02/n02/dch/cfdm/cfdm/core/ __init__.py", line 31, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/n02/n02/shakka/create_IVT_vars.py", line 2, in <module>
    import cf
  File "/home/n02/n02/dch/cf-python/cf/ __init__.py", line 97, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cf v3.14.1 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4'
/work/n02/n02/shakka/cylc-run/u-cz527/app/calc_IVT/bin/main.sh: line 3: cdo: command not found
[FAIL] main.sh <<' __STDIN__'
[FAIL] 
[FAIL] ' __STDIN__' # return-code=127
2023-10-07T07:21:55Z CRITICAL - failed/EXIT

```

App is calc\_IVT and suite is u-cz527.

The pre-script in suite.rc is

```auto
  pre-script = """
                export PATH=/home/n02/n02/dch/cf/bin:$PATH
                export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH
                """
        env-script = """
                eval $(rose task-env)
                module load cdo
                """

```

anything else I should add in there to get netCDF4 to work?

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [8 October 2023 11:02 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/6 "2023-10-08T11:02:32Z")

</div>

Ella

please add `module load netcdf4`

Grenville

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [8 October 2023 13:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/7 "2023-10-08T13:57:42Z")

</div>

Tried adding ‘module load netcdf4’ to the env-script, pre-script and main.sh but still doesn’t seem to want to work. I also get the same error if I try to run create\_IVT\_vars.py directly from the archer command line. Is the cf-python install overwriting the call to load netcdf4 or something?

```auto

Traceback (most recent call last):
  File "/home/n02/n02/dch/cfdm/cfdm/ __init__.py", line 68, in <module>
    import netcdf_flattener
ModuleNotFoundError: No module named 'netcdf_flattener'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/n02/n02/dch/cf-python/cf/ __init__.py", line 95, in <module>
    import cfdm
  File "/home/n02/n02/dch/cfdm/cfdm/ __init__.py", line 70, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cfdm requires the modules cftime, netcdf_flattener. No module named 'netcdf_flattener'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/n02/n02/shakka/create_IVT_vars.py", line 2, in <module>
    import cf
  File "/home/n02/n02/dch/cf-python/cf/ __init__.py", line 97, in <module>
    raise ImportError(_error0 + str(error1))
ImportError: cf v3.14.1 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm requires the modules cftime, netcdf_flattener. No module named 'netcdf_flattener'

cdo (Abort): e[4m/work/n02/n02/shakka/calc_IVT.she[0m -c 19991231T1200Z -s u-cz527  
                             ^ Operator not found
[FAIL] main.sh <<' __STDIN__'
[FAIL] 
[FAIL] ' __STDIN__' # return-code=1
2023-10-08T13:41:55Z CRITICAL - failed/EXIT

```

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 October 2023 14:10 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/8 "2023-10-09T14:10:25Z")

</div>

pl give us read permission on /work/n02/n02/shakka/create\_IVT\_vars.py

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [10 October 2023 08:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/9 "2023-10-10T08:54:06Z")

</div>

Done. Have also give permissions on calc\_IVT.sh.

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [10 October 2023 09:56 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/10 "2023-10-10T09:56:58Z")

</div>

Ella - this looks like an environment problem – remove from your .bashrc  
`export PATH="/home/n02/n02/shakka/.local/bin:$PATH"`

and possibly  
`export PATH="/work/n02/n02/shakka/:$PATH"`

there is no need to module load netcdf4 (that was a mistake on my part)

Grenville

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [10 October 2023 10:16 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/11 "2023-10-10T10:16:50Z")

</div>

I think those lines are the ones you suggested I add to my .bashrc to allow the get\_era\_data app (esp cdsapi) to work - will removing this interfere with that?

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [10 October 2023 10:32 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/12 "2023-10-10T10:32:59Z")

</div>

It looks like you’ll need separate environments for the the various tasks …

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [10 October 2023 10:46 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/13 "2023-10-10T10:46:44Z")

</div>

That’s sensible. Is the preferred method on archer2 to use conda environments?

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [16 October 2023 09:11 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/14 "2023-10-16T09:11:49Z")

</div>

Hi Grenville,

Have created separate conda environments to use cf-python and cdsapi now. But i am now getting an error about a rose-app task call to python2 in the /y07/shared directory:

> /work/y07/shared/umshared/software/rose-pumatest/bin/rose-task-run: line 57: exec: python2: not found  
> /usr/bin/env: ‘python2’: No such file or directory

I can’t see where this is coming from exactly, and I’ve removed all the pre-script and env-script lines except the calls to activate the relevant conda environment and then module load cdo. I’m guessing there’s some rose script that’s still running python 2.7 somewhere?

Any ideas how to resolve this?

Cheers  
Ella

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [17 October 2023 11:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/15 "2023-10-17T11:15:02Z")

</div>

Ella

Please `chmod -R g+rX /home/n02/n02/shakka` on puma2

Grenville

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [17 October 2023 11:21 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/16 "2023-10-17T11:21:36Z")

</div>

what is the suite id?

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [17 October 2023 12:03 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/17 "2023-10-17T12:03:37Z")

</div>

Suite id is u-cz527, have granted permissions.

FYI the two environments work interactively (/work/n02/n02/shakka/ERA5\_env and …/IVT\_env) but I’m just struggling to figure out how best to call them in the apps.

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [17 October 2023 12:32 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/18 "2023-10-17T12:32:21Z")

</div>

Am getting this now:

> CommandNotFoundError: Your shell has not been properly configured to use ‘conda activate’.  
> To initialize your shell, run
> 
> ```
> $ conda init <SHELL_NAME>
> 
> ```
> 
> Currently supported shells are:
> 
> - bash
> - fish
> - tcsh
> - xonsh
> - zsh
> - powershell
> 
> See ‘conda init --help’ for more information and options.
> 
> IMPORTANT: You may need to close and restart your shell after running ‘conda init’.

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [17 October 2023 16:28 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/19 "2023-10-17T16:28:11Z")

</div>

Update: got it to run using #!/bin/bash --login in my .bashrc  
but now the script is not behaving as expected, so digging into that now…

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [17 October 2023 16:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/20 "2023-10-17T16:44:07Z")

</div>

Ella

```auto
[grenvill@puma2 ~]$ cd ~shakka
-bash: cd: /home/n02/n02/shakka: Permission denied

```

[Next page](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155.md?page=2)