what do you load to get cdsapi?
“module load cray-python” - have included it as a pre-script in the definition of HPC_SERIAL and as an init-script in both HPC_SERIAL and get_era_data
silly question perhaps: it’s definitely running that command on archer, not puma right?
Oh, I also had to update the /bin/get_cycle_offset script to use python3 syntax (for printing $CYCLE_OFFSET)
I don’t get cdsapi from cray-python - are you sure you’d not got some other python loaded when it ran on ARCHER ?
grenvill@ln04:~> module load cray-python
grenvill@ln04:~> module list
Currently Loaded Modules:
1) craype-x86-rome 5) xpmem/2.5.2-2.4_3.30__gd0f7936.shasta 9) cray-mpich/8.1.23 13) epcc-setup-env
2) libfabric/1.12.1.2.2.0.0 6) cce/15.0.0 10) cray-libsci/22.12.1.1 14) load-epcc-module
3) craype-network-ofi 7) craype/2.7.19 11) PrgEnv-cray/8.3.3 15) cray-python/3.9.13.1
4) perftools-base/22.12.0 8) cray-dsmml/0.2.2 12) bolt/0.8
grenvill@ln04:~> python
Python 3.9.13 (main, Aug 10 2022, 17:20:06)
[GCC 9.3.0 20200312 (Cray Inc.)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdsapi
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cdsapi'
That’s super weird.
shakka@ln03:~> module unload cray-python
shakka@ln03:~> python /work/n02/n02/shakka/download_startfile.py
Traceback (most recent call last):
File "/work/n02/n02/shakka/download_startfile.py", line 2, in <module>
import cdsapi
**ImportError: No module named cdsapi**
shakka@ln03:~> module load cray-python
shakka@ln03:~> python /work/n02/n02/shakka/download_startfile.py --cycle 20200304T0000Z --remove 20200303T0000Z
20200304T0000Z
Starting MARS retrieval for 2020030400
2023-08-09 11:35:43,573 INFO Welcome to the CDS
2023-08-09 11:35:43,573 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5.1-complete
2023-08-09 11:35:43,684 INFO Request is queued
I can only get cdsapi when cray-python is enabled.
Am going to edit the script and see if it works when I explicitly specify python3…
what do you get for
which python
echo $PYTHONPATH
module list
> /opt/cray/pe/python/3.9.13.1:/work/y07/shared/umshared/lib/python3.9:/work/y07/shared/utils/core/bolt/0.8/modules
>
> Currently Loaded Modules:
> 1) craype-x86-rome 5) xpmem/2.5.2-2.4_3.30__gd0f7936.shasta 9) cray-mpich/8.1.23 13) epcc-setup-env
> 2) libfabric/1.12.1.2.2.0.0 6) cce/15.0.0 10) cray-libsci/22.12.1.1 14) load-epcc-module
> 3) craype-network-ofi 7) craype/2.7.19 11) PrgEnv-cray/8.3.3 15) cray-python/3.9.13.1
> 4) perftools-base/22.12.0 8) cray-dsmml/0.2.2 12) bolt/0.8
I suspect it’s something to do with python2/3, as I get a python error about the format of the print statement in get_cycle_offset if I leave L71 as print cycle_offset, but if I change to python 3 syntax (ie. print(cycle_offset) ), I get the following error:
usage: get_cycle_offset [-h] [–curr_date CURR_DATE]
[–dates DATES [DATES …]]
get_cycle_offset: error: argument --dates: expected at least one argument
The offending line in suite.rc seems to be:
CYCLE_OFFSET=$($CYLC_SUITE_RUN_DIR/bin/get_cycle_offset --curr_date $CURR_DATE --dates {{DATES|join(" ")}})
which I added to get_era_data to make $CYCLE_OFFSET available as an environment variable that I can feed to download_startfile.py .
Is there another way I can get $CYCLE_OFFSET to work…?
Ella
What’s happening here in your .bashrc: export PATH=“/home/n02/n02/shakka/.local/bin:$PATH”
I don’t have permission to see in.
You probably need to add that to the get_era_data pre-script
Grenville
Hmm, not sure what’s going on there, but this is the full path:
/home/n02/n02/shakka/.local/bin:/opt/cray/pe/python/3.9.13.1/bin:/work/n02/n02/shakka:/home/n02/n02/shakka/.local/bin:/work/y07/shared/utils/core/bolt/0.8/bin:/work/y07/shared/utils/core/bin:/opt/cray/pe/mpich/8.1.23/ofi/crayclang/10.0/bin:/opt/cray/pe/mpich/8.1.23/bin:/opt/cray/pe/craype/2.7.19/bin:/opt/cray/pe/cce/15.0.0/binutils/x86_64/x86_64-pc-linux-gnu/bin:/opt/cray/pe/cce/15.0.0/binutils/cross/x86_64-aarch64/aarch64-linux-gnu/…/bin:/opt/cray/pe/cce/15.0.0/utils/x86_64/bin:/opt/cray/pe/cce/15.0.0/bin:/opt/cray/pe/cce/15.0.0/cce-clang/x86_64/bin:/opt/cray/pe/perftools/22.12.0/bin:/opt/cray/pe/papi/6.0.0.17/bin:/opt/cray/libfabric/1.12.1.2.2.0.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/opt/cray/pe/bin:/work/y07/shared/umshared/software/bin:/work/y07/shared/umshared/bin:/work/y07/shared/utils/core/python/miniconda2/bin
… So I should add the export PATH=“/home/n02/n02/shakka/.local/bin:$PATH” line to my get_era_data pre-script?
pl allow us to read /home/n02/n02/shakka/.local/bin
Done!
(adding some characters)
grenvill@ln04:/home/n02/n02/dcase/cylc-run/u-cw083> cd /home/n02/n02/shakka/.local/bin
-bash: cd: /home/n02/n02/shakka/.local/bin: Permission denied
chmod -R g+rX /home/n02/n02/shakka (will let us read everything)
Strange, thought that’s what I’d done, but evidently not. Should hopefully work now.
Well, I’ve failed to get cdsapi - I have used your basrc and bash_profile
please could you
python
import cdsapi
print(cdsapi.__file__)
aha, it’s local: /home/n02/n02/shakka/.local/lib/python3.9/site-packages/cdsapi/init.py
I guess I must’ve installed it within my cray-python environment?
Ella
You must have done some cdsapi set up - you have hidden config files - I don’t understand your environment.
I got cdsapi by doing
export PYTHONPATH=/home/n02/n02/shakka/.local/lib/python3.9/site-packages:$PYTHONPATH
but needed a .cdsapirc which I don’t have
try adding this to the pre-script
( or
export PATH="/home/n02/n02/shakka/.local/bin:$PATH"
export PATH="/work/n02/n02/shakka/:$PATH"
but that didn’t work for me, I still can’t see where your current set up adds the module)
Grenville
1 Like
Ah ok, I have a .cdsapirc file with my authentication details. I’m now able to run ‘module load cray-python’ it seems, but I’m still getting an error related to the $CYCLE_OFFSET variable:
> usage: get_cycle_offset [-h] [--curr_date CURR_DATE]
> [--dates DATES [DATES ...]]
> get_cycle_offset: error: argument --dates: expected at least one argument
I’m guessing {{DATES|join(" ")}} is empty - what are DATES?
yes they are. It’s the line used in the housekeeping script as a call to get_cycle_offset to determine which cycle files to delete. I foolishly assumed I could call it here in the same way to get the current cycle. Interestingly the $CURR_DATE works as I can print that out to the .out file, so I could potentially do something clever with that in my python script…