# Install of convsh on archer2? **URL:** \ **Category:** General\ **Tags:** ARCHER2\ **Created:** [3 October 2023 14:13 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155 "2023-10-03T14:13:31Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 07:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/21 "2023-10-18T07:57:29Z")
woops, sorry. Fixed now. Although the conda environments I’m trying to use are on /work so that the compute nodes can see them. ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 08:03 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/22 "2023-10-18T08:03:44Z")
ah, just realised… I’m still on pumanew, now puma2. It’s on my to-do list to swap over today ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [18 October 2023 08:10 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/23 "2023-10-18T08:10:49Z")
I’m glad you are fixing things - I can’t keep up with the pace of change ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 09:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/24 "2023-10-18T09:55:23Z")
I’ve got a bit stuck on the last bit of the conversion to puma2 actually- struggling to figure out which host select line I should change (I don’t have site/archer2.rc files, only site/ncas-cray-ex/suite-adds.rc etc) should I be changing {% set HPC\_HOST = “[login.archer2.ac.uk](http://login.archer2.ac.uk)” %} or something later like [[[remote]]] host = {{HPC\_HOST}} ? ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [18 October 2023 10:21 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/25 "2023-10-18T10:21:57Z")
Try `{% set HPC_HOST = "archer2" %}` then `host = $(rose host-select {{HPC_HOST}})` ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 10:38 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/26 "2023-10-18T10:38:09Z")
Thanks Grenville, I just discovered [this ticket](https://cms-helpdesk.ncas.ac.uk/t/issue-with-suite-add-rc-file-for-puma2-line-starting-with-set-hpc-host/1191) and followed these instructions. I’m still getting the connection timed out error though, so trying to figure out what I’ve done wrong… ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [18 October 2023 11:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/27 "2023-10-18T11:55:11Z")
connection timed out error ? ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 12:03 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/28 "2023-10-18T12:03:45Z")
I get: > [FAIL] ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n [login2.archer2.ac.uk](http://login2.archer2.ac.uk) env\ ROSE\_VERSION=2019.01.3\ CYLC\_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-cy223\ –run=restart\ –remote=uuid=646874ff-4bde-4aca-8b01-03f2708ac8eb,now-str=20231018T120218Z # return-code=255, stderr= > [FAIL] ssh: connect to host [login2.archer2.ac.uk](http://login2.archer2.ac.uk) port 22: Connection timed out …which according to the changeover FAQ/troubleshooting means I have incorrectly updated the ‘host = {{HOST\_HPC}}’ lines somewhere… ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [18 October 2023 12:22 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/29 "2023-10-18T12:22:04Z")
pl Please chmod -R g+rX /home/n02/n02/shakka on puma2 ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 12:26 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/30 "2023-10-18T12:26:36Z")
> [@grenville](#): > > chmod -R g+rX /home/n02/n02/shakka that should be done! ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 12:36 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/31 "2023-10-18T12:36:10Z")
A strange development is that my other suites (e.g. u-cz527) _have_ restarted - I am using exactly the same suite-adds.rc file there, but u-cy223 seems reluctant to restart. ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [18 October 2023 12:49 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/32 "2023-10-18T12:49:17Z")
in suite.rc u-cy223 delete 2x ```auto [[[remote]]] host=login2.archer2.ac.uk ``` ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 13:00 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/33 "2023-10-18T13:00:21Z")
This is what we added to get the archiving steps to run on a specific node, right? Will the archiving step break if I do this? ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [18 October 2023 13:21 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/34 "2023-10-18T13:21:44Z")
no - rose host-select takes care of the polling correctly ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [18 October 2023 13:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/35 "2023-10-18T13:54:26Z")
Great, it runs once I change that. Thanks! Now back to the original problem… I’m now discovering that there is a variable selection error in my cdo script, which causes the calc\_IVT app to say it’s succeeded but without actually producing the output I want. … investigating now. ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [19 October 2023 10:23 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/36 "2023-10-19T10:23:24Z")
Right, so. Here’s a more detailed description of the problem. When I add the `conda activate /work/n02/n02/shakka/IVT_env` to the pre-script in suite.rc, I get an error related to python2, because it seems that the rose app execution scripts are written in python2 (I hope this might change in future?!) > /work/y07/shared/umshared/metomi/rose-2019.01.3/bin/rose-task-run: line 57: exec: python2: not found > /usr/bin/env: ‘python2’: No such file or directory When I put it in the env-script section I get a submit-retrying error and no .out or .err output to diagnose. To get around this, I have tried to add the `conda activate` command in the bin/main.sh script. But it doesn’t seem to have the same effect as if I run exactly the same commands directly from the command line: For reference, I’m doing: ```auto conda activate /work/n02/n02/shakka/IVT_env python /work/n02/n02/shakka/create_IVT_vars.py --cycle "$CYLC_TASK_CYCLE_POINT" --suite "u-cy223" bash /work/n02/n02/shakka/calc_IVT.sh -c "$CYLC_TASK_CYCLE_POINT" -s "u-cy223" ``` But the task finishes with ‘succeeded’, and prints the following into the .err file, which references the default base conda environment in /work/y07/: ```auto Traceback (most recent call last): File "/work/y07/shared/utils/core/python/miniconda2/bin/conda-env", line 6, in from conda_env.cli.main import main File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda_env/cli/main.py", line 13, in import conda.exports # noqa File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/exports.py", line 25, in from . import plan # NOQA File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/plan.py", line 25, in from .core.index import LAST_CHANNEL_URLS, _supplement_index_with_prefix File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/index.py", line 10, in from .package_cache_data import PackageCacheData File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/package_cache_data.py", line 15, in from .path_actions import CacheUrlAction, ExtractPackageAction File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/path_actions.py", line 30, in from ..gateways.connection.download import download File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/gateways/connection/ __init__.py", line 21, in from requests import ConnectionError, HTTPError, Session File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/requests/ __init__.py", line 43, in import urllib3 File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/urllib3/ __init__.py", line 36 f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. " ^ SyntaxError: invalid syntax ``` …so this suggests that my attempts to call my conda env are not working: can you help me understand why the app is not activating it? ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [19 October 2023 15:16 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/37 "2023-10-19T15:16:42Z")
It looks like `create_IVT_vars.py` just needs cf - why not simply do what we do to get cf ( i don’t know how conda environments work in cylc (not well it seems)) ``` pre-script = """ export PATH=/home/n02/n02/dch/cf/bin:$PATH export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH """ ``` ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [19 October 2023 15:28 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/38 "2023-10-19T15:28:23Z")
Thanks, I’ll try this and hope it doesn’t conflict with the packages I need for get\_era\_data Will report back! ---
**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\ **Post date:** [19 October 2023 19:37 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/39 "2023-10-19T19:37:47Z")
Tried the above, but am getting the following error complaining about the lack of netcdf4 (which I think is the same place we got to [here](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/6) ```auto Traceback (most recent call last): File "/work/y07/shared/utils/core/python/miniconda2/bin/conda-env", line 6, in from conda_env.cli.main import main File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda_env/cli/main.py", line 13, in import conda.exports # noqa File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/exports.py", line 25, in from . import plan # NOQA File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/plan.py", line 25, in from .core.index import LAST_CHANNEL_URLS, _supplement_index_with_prefix File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/index.py", line 10, in from .package_cache_data import PackageCacheData File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/package_cache_data.py", line 15, in from .path_actions import CacheUrlAction, ExtractPackageAction File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/path_actions.py", line 30, in from ..gateways.connection.download import download File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/gateways/connection/ __init__.py", line 21, in from requests import ConnectionError, HTTPError, Session File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/requests/ __init__.py", line 43, in import urllib3 File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/urllib3/ __init__.py", line 36 f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. " ^ SyntaxError: invalid syntax Traceback (most recent call last): File "/home/n02/n02/dch/cfdm/cfdm/core/ __init__.py", line 29, in import netCDF4 File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/netCDF4/ __init__.py", line 3, in from ._netCDF4 import * ModuleNotFoundError: No module named 'netCDF4._netCDF4' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/n02/n02/dch/cf-python/cf/ __init__.py", line 95, in import cfdm File "/home/n02/n02/dch/cfdm/cfdm/ __init__.py", line 44, in from . import core File "/home/n02/n02/dch/cfdm/cfdm/core/ __init__.py", line 31, in raise ImportError(_error0 + str(error1)) ImportError: cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/work/n02/n02/shakka/create_IVT_vars.py", line 2, in import cf File "/home/n02/n02/dch/cf-python/cf/ __init__.py", line 97, in raise ImportError(_error0 + str(error1)) ImportError: cf v3.14.1 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4' cdo(3) selname: Open failed on >dp_prof.nc< No such file or directory cdo(3) selname: Open failed on >dp_prof.nc< No such file or directory cdo(3) selname: Open failed on >dp_prof.nc< No such file or directory cdo(3) selname: Open failed on >dp_prof.nc< No such file or directory ``` Reading back, this is the same thing as we tried earlier which brought us to the conclusion that I should use separate conda environments… …so I guess the question is probably going to have to be how to get conda environments to work with cylc? ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [20 October 2023 09:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155/40 "2023-10-20T09:14:05Z")
Ella This is the problem: ``` [[HPC_SERIAL]] inherit = None, HOST_HPC init-script = """ export PYTHONPATH=/home/n02/n02/shakka/.local/lib/python3.9/site-packages:$PYTHONPATH module load cray-python """ ``` this loads stuff that interferes downstream - I’d set up the environment individually for each task ( maybe rationalize later.) Please take a look at my copy of u-cy223 – the netcdf4 problem goes away (I currently have a cfunits error that we’re looking at) We set PATH (and PYTHONPATH occasionally) rather than conda activate - you probably could `export PATH= /work/n02/n02/shakka/IVT_env/bin:$PATH` (I’ve not tried this) Grenville [Previous page](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155.md?page=1) [Next page](https://cms-helpdesk.ncas.ac.uk/t/install-of-convsh-on-archer2/1155.md?page=3)