there are a slew of cylc environment variables here 9. Suite Configuration — The Cylc Suite Engine 7.8.8 documentation
any help
1 Like
Aha, okay. I’ve changed the python script to use $CYLC_CURRENT_CYCLE_POINT and use python to figure out which files to download ahead of time and which to remove. Hopefully that should work.
Presumably I need to tell the app to inherit the variable somewhere, or are they already loaded in in the cylc environment?
… although I notice archer /work isn’t running properly at the moment, which explains why my job was endlessly hanging
$CYLC_CURRENT_CYCLE_POINT should be available by default
It didn’t seem to be available (maybe it was the way i was specifying it), so i changed it to use $CURR_DATE instead.
But now I’m getting the cdsapi error again, which I’ve realised I’m getting because cdsapi is installed on /home and it isn’t visible to the compute nodes.
Can I / someone install cdsapi on the /work filesystem?
Yes - I should have caught that – constant source of error!
Run it on the login node instead - if it’s not resource hungry?
1 Like
I managed to get it to work and get beyond the initial cdsapi error by adding the path to my local install of cdsapi to the init-script for HPC_SERIAL, but it sounds like running on the login nodes might be useful anyway, especially if anyone else wants to use the app eventually.
Would I set that up by adding a line to [[HPC_SERIAL]] in /site/ncas-cray-ex/suite-adds.rc ?
For future reference, in case you should need to do something similar again or anyone else reads this, the cylc environment variables should be available in a python script by using os.environ['CYLC_CURRENT_CYCLE_POINT']
Thanks for this Ros. I tried this but get the following error:
Traceback (most recent call last):
File “/work/n02/n02/shakka/test.py”, line 6, in
cycle = os.environ[‘CYLC_CURRENT_CYCLE_POINT’]
File “/opt/cray/pe/python/3.9.13.1/lib/python3.9/os.py”, line 679, in getitem
raise KeyError(key) from None
KeyError: ‘CYLC_CURRENT_CYCLE_POINT’
/work/n02/n02/shakka/cylc-run/u-cy223/app
Oh I see the problem CYLC_CURRENT_CYCLE_POINT is not a valid cylc variable - it’s CYLC_TASK_CYCLE_POINT
Cheers,
Ros.
aha! Yes this seems to work… and defining within the python script saves me feeding it command line inputs.
Update: have successfully managed to get my app to run - thanks for your help! Am now going to move onto the next part of the task, which is to set up the transfer of output files to jasmin…
Cheers
Ella
I’m looking at the docs for rose-arch now, and wondering where to put the step in the suite graph. I have kind of assumed the suite-graph/lam-fcst.rc would be the best place (as archiving is already referenced there).
Also, is there a suite id I can look at that has archiving set up (from archer2 to jasmin ideally, but I know this might be a lot to ask!) ?
Cheers
Ella
Hi Ella
It’s not easy to follow the jinja in u-cy223 – I’m only guessing where to put an archive task, but in
suite-graph/lam-fcst.rc and suite-graph/dm.rc , adding:
archive => housekeep_cycle
makes a graph like:
install_glm_startdata => glm_um_recon1 => glm_um_fcst_000 => glm_um_fcst_001 => glm_um_fcst_002 => glm_um_fcst_003 => glm_um_fcst_004 => glm_um_fcst_005 => archive => housekeep_cycle
Arctic_11km_ga7_24-36_um_recon => Arctic_11km_ga7_24-36_um_fcst_000 => Arctic_11km_ga7_24-36_um_fcst_001 => Arctic_11km_ga7_24-36_um_fcst_002 => Arctic_11km_ga7_24-36_um_fcst_003 => Arctic_11km_ga7_24-36_um_fcst_004 => Arctic_11km_ga7_24-36_um_fcst_005 => archive => housekeep_cycle
Antarctic_11km_ga7_24-36_um_recon => Antarctic_11km_ga7_24-36_um_fcst_000 => Antarctic_11km_ga7_24-36_um_fcst_001 => Antarctic_11km_ga7_24-36_um_fcst_002 => Antarctic_11km_ga7_24-36_um_fcst_003 => Antarctic_11km_ga7_24-36_um_fcst_004 => Antarctic_11km_ga7_24-36_um_fcst_005 => archive => housekeep_cycle
We don’t use the built in archive app for climate suites so I can not point to a suite that uses the built in app for archiving to JASMIN.
Grenville
Hi Grenville,
Thanks for this. I’ll add it in here.
So you’re saying there are no suites currently archiving archer → jasmin? That bodes well!
I’m struggling to decipher the archiving rename syntax. I’d like to be able to rename the files produced by the suite according to the region name and cycle point before transferring them because I’m using the same suite to produce output for Antarctic and Arctic. I presume I can use ROSE variables like $ROSE_SUITE_NAME for this?
Would it look something like this?
[arch:field.nc/]
command-format=rsync %(sources)s %(target)s
rename-format=%(cycle)s_${NAME}_p%(tag)s
rename-parser=^.(?P..nc)$
source=*.nc
source-prefix=$RUN_DIR
Cheers
Ella
I’m going to start a new thread on the archiving, because it seems to make more sense to do this separately… hope that’s all good!