woops, sorry. Fixed now. Although the conda environments I’m trying to use are on /work so that the compute nodes can see them.
ah, just realised… I’m still on pumanew, now puma2. It’s on my to-do list to swap over today
I’m glad you are fixing things - I can’t keep up with the pace of change
I’ve got a bit stuck on the last bit of the conversion to puma2 actually- struggling to figure out which host select line I should change (I don’t have site/archer2.rc files, only site/ncas-cray-ex/suite-adds.rc etc)
should I be changing
{% set HPC_HOST = “login.archer2.ac.uk” %}
or something later like
[[[remote]]]
host = {{HPC_HOST}}
?
Try
{% set HPC_HOST = "archer2" %}
then
host = $(rose host-select {{HPC_HOST}})
Thanks Grenville, I just discovered this ticket and followed these instructions.
I’m still getting the connection timed out error though, so trying to figure out what I’ve done wrong…
connection timed out error ?
I get:
[FAIL] ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n login2.archer2.ac.uk env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-cy223\ –run=restart\ –remote=uuid=646874ff-4bde-4aca-8b01-03f2708ac8eb,now-str=20231018T120218Z # return-code=255, stderr=
[FAIL] ssh: connect to host login2.archer2.ac.uk port 22: Connection timed out
…which according to the changeover FAQ/troubleshooting means I have incorrectly updated the ‘host = {{HOST_HPC}}’ lines somewhere…
pl Please chmod -R g+rX /home/n02/n02/shakka on puma2
that should be done!
A strange development is that my other suites (e.g. u-cz527) have restarted - I am using exactly the same suite-adds.rc file there, but u-cy223 seems reluctant to restart.
in suite.rc u-cy223 delete 2x
[[[remote]]]
host=login2.archer2.ac.uk
This is what we added to get the archiving steps to run on a specific node, right? Will the archiving step break if I do this?
no - rose host-select takes care of the polling correctly
1 Like
Great, it runs once I change that. Thanks! Now back to the original problem…
I’m now discovering that there is a variable selection error in my cdo script, which causes the calc_IVT app to say it’s succeeded but without actually producing the output I want.
… investigating now.
Right, so. Here’s a more detailed description of the problem.
When I add the conda activate /work/n02/n02/shakka/IVT_env to the pre-script in suite.rc, I get an error related to python2, because it seems that the rose app execution scripts are written in python2 (I hope this might change in future?!)
/work/y07/shared/umshared/metomi/rose-2019.01.3/bin/rose-task-run: line 57: exec: python2: not found
/usr/bin/env: ‘python2’: No such file or directory
When I put it in the env-script section I get a submit-retrying error and no .out or .err output to diagnose.
To get around this, I have tried to add the conda activate command in the bin/main.sh script. But it doesn’t seem to have the same effect as if I run exactly the same commands directly from the command line:
For reference, I’m doing:
conda activate /work/n02/n02/shakka/IVT_env
python /work/n02/n02/shakka/create_IVT_vars.py --cycle "$CYLC_TASK_CYCLE_POINT" --suite "u-cy223"
bash /work/n02/n02/shakka/calc_IVT.sh -c "$CYLC_TASK_CYCLE_POINT" -s "u-cy223"
But the task finishes with ‘succeeded’, and prints the following into the .err file, which references the default base conda environment in /work/y07/:
Traceback (most recent call last):
File "/work/y07/shared/utils/core/python/miniconda2/bin/conda-env", line 6, in <module>
from conda_env.cli.main import main
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda_env/cli/main.py", line 13, in <module>
import conda.exports # noqa
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/exports.py", line 25, in <module>
from . import plan # NOQA
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/plan.py", line 25, in <module>
from .core.index import LAST_CHANNEL_URLS, _supplement_index_with_prefix
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/index.py", line 10, in <module>
from .package_cache_data import PackageCacheData
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/package_cache_data.py", line 15, in <module>
from .path_actions import CacheUrlAction, ExtractPackageAction
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/path_actions.py", line 30, in <module>
from ..gateways.connection.download import download
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/gateways/connection/__init__.py", line 21, in <module>
from requests import ConnectionError, HTTPError, Session
File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/requests/__init__.py", line 43, in <module>
import urllib3
File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/urllib3/__init__.py", line 36
f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
^
SyntaxError: invalid syntax
…so this suggests that my attempts to call my conda env are not working: can you help me understand why the app is not activating it?
It looks like create_IVT_vars.py just needs cf - why not simply do what we do to get cf ( i don’t know how conda environments work in cylc (not well it seems))
pre-script = """
export PATH=/home/n02/n02/dch/cf/bin:$PATH
export PYTHONPATH=/home/n02/n02/dch/cfdm:/home/n02/n02/dch/cf-python:$PYTHONPATH
"""
Thanks, I’ll try this and hope it doesn’t conflict with the packages I need for get_era_data
Will report back!
Tried the above, but am getting the following error complaining about the lack of netcdf4 (which I think is the same place we got to here
Traceback (most recent call last):
File "/work/y07/shared/utils/core/python/miniconda2/bin/conda-env", line 6, in <module>
from conda_env.cli.main import main
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda_env/cli/main.py", line 13, in <module>
import conda.exports # noqa
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/exports.py", line 25, in <module>
from . import plan # NOQA
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/plan.py", line 25, in <module>
from .core.index import LAST_CHANNEL_URLS, _supplement_index_with_prefix
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/index.py", line 10, in <module>
from .package_cache_data import PackageCacheData
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/package_cache_data.py", line 15, in <module>
from .path_actions import CacheUrlAction, ExtractPackageAction
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/core/path_actions.py", line 30, in <module>
from ..gateways.connection.download import download
File "/work/y07/shared/utils/core/python/miniconda2/lib/python2.7/site-packages/conda/gateways/connection/__init__.py", line 21, in <module>
from requests import ConnectionError, HTTPError, Session
File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/requests/__init__.py", line 43, in <module>
import urllib3
File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/urllib3/__init__.py", line 36
f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
^
SyntaxError: invalid syntax
Traceback (most recent call last):
File "/home/n02/n02/dch/cfdm/cfdm/core/__init__.py", line 29, in <module>
import netCDF4
File "/home/n02/n02/shakka/.local/lib/python3.9/site-packages/netCDF4/__init__.py", line 3, in <module>
from ._netCDF4 import *
ModuleNotFoundError: No module named 'netCDF4._netCDF4'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/n02/n02/dch/cf-python/cf/__init__.py", line 95, in <module>
import cfdm
File "/home/n02/n02/dch/cfdm/cfdm/__init__.py", line 44, in <module>
from . import core
File "/home/n02/n02/dch/cfdm/cfdm/core/__init__.py", line 31, in <module>
raise ImportError(_error0 + str(error1))
ImportError: cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/n02/n02/shakka/create_IVT_vars.py", line 2, in <module>
import cf
File "/home/n02/n02/dch/cf-python/cf/__init__.py", line 97, in <module>
raise ImportError(_error0 + str(error1))
ImportError: cf v3.14.1 requires the modules numpy, netCDF4, cftime, cfunits, cfdm, psutil, dask, packaging. cfdm.core requires the modules numpy, netCDF4, packaging. No module named 'netCDF4._netCDF4'
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
cdo(3) selname: Open failed on >dp_prof.nc<
No such file or directory
Reading back, this is the same thing as we tried earlier which brought us to the conclusion that I should use separate conda environments…
…so I guess the question is probably going to have to be how to get conda environments to work with cylc?
Ella
This is the problem:
[[HPC_SERIAL]]
inherit = None, HOST_HPC
init-script = """
export PYTHONPATH=/home/n02/n02/shakka/.local/lib/python3.9/site-packages:$PYTHONPATH
module load cray-python
"""
this loads stuff that interferes downstream - I’d set up the environment individually for each task ( maybe rationalize later.)
Please take a look at my copy of u-cy223 – the netcdf4 problem goes away (I currently have a cfunits error that we’re looking at)
We set PATH (and PYTHONPATH occasionally) rather than conda activate - you probably could
export PATH= /work/n02/n02/shakka/IVT_env/bin:$PATH
(I’ve not tried this)
Grenville
1 Like