Pptransfer error

1

Hi CMS,

I’m running a fully coupled UKESM1.1 suite u-de744 for one year on Archer2 and have run into an error with pptransfer.

The error info is below:

[WARN] [SUBPROCESS]: Command: globus-url-copy -vb -cd -r -cc 4 -verify-checksum -sync -cred /work/n02/n02/jingjin562/cred.jasmin /work/n02/n02/jingjin562/archive/ gsiftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/terrafirma/jjin/archer2/u-de744/20060101T0000Z/
[SUBPROCESS]: Error = 1:

error: globus_ftp_client: the operation was aborted

[WARN] Transfer command failed: globus-url-copy -vb -cd -r -cc 4 -verify-checksum -sync -cred /work/n02/n02/jingjin562/cred.jasmin /work/n02/n02/jingjin562/archive/ gsiftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/terrafirma/jjin/archer2/u-de744/20060101T0000Z/
[ERROR] transfer.py: Unknown Error - Return Code=1
[FAIL] Command Terminated
[FAIL] Terminating PostProc…
[FAIL] transfer.py <<‘STDIN’
[FAIL]
[FAIL] ‘STDIN’ # return-code=1
2024-05-02T09:57:38Z CRITICAL - failed/EXIT

This error is robus when the job is submitted to a queue. However, when I run command “globus-url-copy -vb -cd -r -cc 4 -verify-checksum -sync -cred /work/n02/n02/jingjin562/cred.jasmin /work/n02/n02/jingjin562/archive/ gsiftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/terrafirma/jjin/archer2/u-de744/20060101T0000Z/” on a login node, the pptransfer works fine. All files are copied to my jasmin dirctory.

How can I fix this error?

Cheers,
Jing

2

Hi Jing,

There is a problem with your suite setup it is dumping all files from every cycle into the /work/n02/n02/jingjin562/archive directory which means the gridftp command will never complete as files are continually being written. The directory structure should be /work/n02/n02/jingjin562/archive/u-de744/<cycle-point>.

I’ll take a look at your suite and get back to you.

Cheers,
Ros.

3

Hi Jing,

Ah, I see, it’s using TerraFirma postprocessing which is a special version of postproc. In postproc → Post processing - common settings → ARCHER2-JASMIN Archiving please set archive_root_path = $ROSE_DATAC.

PPtransfer should then work fine from the next coupled cycle that is run. Once the cycle that is currently running has completed postproc. You will then have to manually gridftp all the files under /work/n02/n02/jingjin562/archive to JASMIN.

To ensure data on ARCHER2 is removed once it has been successfully transferred to JASMIN you will need to add the following into the housekeeping app. In the rose app/housekeeping/rose-app.conf file under [[prune] add the line:

prune{share/cycle}=-P6M

Regards,
Ros.

4

I probably also should have said it might just be easiest to make the 2 changes and then start the suite again from the beginning again.

5

Hi Ros,

Thanks for your help. I’ve made the changes and the suite works well.

Cheers,
Jing