I want to try Cylc on ARCHER2 (I think ideally not with Rose – I just want to be able to run programs that are in a workflow defined by Cylc.)
May I ask:
Do Cylc and Archer2 yet work well together? (I note questions about the need of Cylc to use the home directory in a Cylc ticket, but was unclear if it was resolved)
is there guidance as to how to start running simple tests on Archer2 with Cylc (maybe from PUMA)?
Thanks; regards
Mike
Hi Mike,
So yes Rose/Cylc (e.g. UM workflow) is all working between PUMA/ARCHER2.
Is there a particular reason why you don’t wish to use Rose? As you allude to Cylc does indeed inherently use home directory to locate the suite’s running files. For Rose/Cylc on ARCHER2 this has been resolved by configuring Rose to symlink the top-level cylc-run suite directory from $HOME to $DATADIR and to configure Cylc to redefine the location of the cylc suite run directory.
Cylc itself doesn’t have the ability to specify symlinking of the $HOME/cylc-run/<suiteid> directory so, whilst not impossible, it is not as straightforward to run pure cylc suites on ARCHER2. Before running any suite you would need to manually create the $DATADIR/cylc-run/<suite-id> and symlink to it from $HOME. I have only run a simple cylc suite this way, as proof of concept, so can’t guarantee that there will be no other issues.
In terms of general environment setup to run Rose/Cylc suites on ARCHER2 from PUMA we have some instructions on our website for creating the special ssh-key required to connect from PUMA to ARCHER2: http://cms.ncas.ac.uk/wiki/Archer2/SshAgentSetup
You will also need to add the following to your ~/.bash_profile on ARCHER2:
. /work/y07/shared/umshared/bin/rose-um-env
I do have the odd noddy suite (both with and without Rose) I can share that runs a simple job on the ARCHER2 compute nodes if helpful. Otherwise for general Rose & Cylc Tutorials I would suggest the Rose User Guide
Hope that helps,
Regards,
Ros
Thanks Ros!
I’d appreciate access to your “noddy suites” with and without Rose.
My answer to the “why not Rose” question may display ignorance: we are wanting to do a cyclical workflow with programs and with MPI models that are not known to Rose - or yet used in Cylc - so I imagined just using Cylc was simpler… happy to be corrected!
Regards
Mike
Hi Mike,
After trying a few more things out, for now, we think it would be easier to run the pure cylc suites from the ARCHER2 login nodes. This means you won’t need to worry about symlinking directories before running suites, etc. To do this you will need to configure a few things on ARCHER2:
Once logged onto ARCHER2 switch $HOME to point to /work directory:
export HOME=/work/n02/n02/<username>
Setup the cylc environment:
mkdir ~/.cylc # This should now be under /work
cp /work/n02/n02/ros/.cylc/global.rc ~/.cylc
I have 2 test suites under /work/n02/n02/ros/cylc-suites/ which you can copy. They should both run without any changes.
cd cylc-suites/oneoff
cylc register oneoff
cylc run oneoff
You will also find a load of example suites in the cylc install directory which you might find useful: /work/y07/shared/umshared/software/cylc-7.8.7/etc/examples. I’m sure you’ve already found it, but there’s a basic cylc tutorial in cylc user guide too. Enjoy! 
Regards
Ros.
Hi Ros
I appreciate your effort in these responses. I am getting a complaint at the content of global.rc and wonder if a bit more setup is needed… such as site.rc or conf?
When running the “cylc register oneoff” response is
oneoff> cylc register oneoff
ERROR: bad user config /work/n02/n02/mjmn02/.cylc/global.rc:
parsec.validate.IllegalItemError: Illegal item: communication
where:
pwd /work/n02/n02/mjmn02/cylc-suites/oneoff
echo $HOME
/work/n02/n02/mjmn02
~/.cylc/global.rc is your one,.
Thanks
Mike
…
Removing the [commnunication]
http
lines from global.rc cleared the above…
Now I can submit the suite with:
cylc-suites> cylc run oneoff ./oneoff
(also setting my own account code in the suite)…
so ignore this for now, please Ros,… untiil an update follows!
Hi Mike
What version of cylc are you running – it feels like 6.11.4 - Ros was testing with 7.8 – probably best to use the later version.
Grenville
Good spot!!
Many thanks, Grenville
(for any readers interested, to get the later cylc, just:
export PATH=/work/y07/shared/umshared/software/cylc-7.8.7/bin:$PATH
and then I could register and run the suite runs as Ros said.)
Mike
or just edit your ARCHER2 .bash_profile to include
export CYLC_VERSION=7.8.7
and (possibly)
export ROSE_VERSION=2019.01.3
export FCM_VERSION=2017.10.0