Post processing on JASMIN, controlled from PUMA

1

Please excuse this naive question - but is it possible to submit tasks from a cylc workflow running on PUMA to JASMIN? I want to add post-processing steps to a predominately ARCHER2 based workflow, in which we already transfer the data to JASMIN. Are there any example workflows that do this that I could copy from?

2

HI Doug,

Yes this is certainly possible. We have suites on PUMA that submit tasks to JASMIN both on the sci nodes and LOTUS.

My suite u-cy010 has an example submission to a jasmin sci node in the jdma.rc file. I have the suite checked out at ~ros/roses/u-cy010

If you’re wanting to submit to run on LOTUS, I can dig out an example. Let me know.

In order to submit to JASMIN you need to set up ssh tunnelling & multiplexing for it to work. Details on how to set this up can be found here: Setup connection to JASMIN sci nodes

Cheers,
Ros.

1 Like
3

Hi Ros,

Thanks for these pointers! I’ll have a look at your suite, and see if I can copy the process to our suite too.

cheers,
Doug

4

hi Ros,

I’ve some trouble getting this working.

I note that the ROSE version expected is 2019.01.3

ssh -oBatchMode=yes -n dlowe@sci2.jasmin.ac.uk env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ \'\"$

but on JASMIN the default version is now 2019.01.8.

Will that cause any problems?

5

Hi Doug,

That’s fine. The 2019.01.3 version is still installed and available for use on JASMIN. We have this in all our suites.

[INFO] 2023-07-24T17:08:51+0100 ssh -oBatchMode=yes -n sci2.jasmin.ac.uk env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12

What problem are you seeing?

Cheers,
Ros.

6

hi Ros,

That’s good to know.

I don’t quite know what the problem is. It seems to be that setup doesn’t happen, but I don’t get a useful (or, indeed, any) error message. I’ll look through your suite setup again, and see if I can find what I’ve missed.

thanks,
Doug

7

Just to update on this - it was an ssh error: because I’d not interactively ssh’d to sci2.jasmin.ac.uk (only to login.jasmin.ac.uk, then onwards from there) from puma, the ssh key verification failed.

Is it possible to get standard error to pipe to the logs when starting a suite in rosie? If that information had been available I’d have realised my error a lot earlier.

8

Hi Doug,

It should throw login errors to the command line output when the suite starts up.

The complications with JASMIN is that if you haven’t started up the connection first it usually just hangs trying to connect through.

Glad you solved the problem.

Cheers,
Ros.