# Problem with pptransfer from ARCHER2 -\> JASMIN **URL:** \ **Category:** Uncategorized\ **Tags:** ARCHER2\ **Created:** [19 January 2024 17:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308 "2024-01-19T17:14:19Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![aschurer](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@aschurer](https://cms-helpdesk.ncas.ac.uk/u/aschurer)\ **Post date:** [19 January 2024 17:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/1 "2024-01-19T17:14:19Z")
Hi, I am running a UKESM1.1 job (u-dc258) on ARCHER2 (PUMA2) and am trying to transfer data to JASMIN. I was having issues with the pptransfer. I’ve have been trying with the old rsync way which I found to work intermittently. So I have tried changing to using GridFTP following this guidance: > **[Configuring PPTransfer](https://cms.ncas.ac.uk/unified-model/pptransfer/)** > > The now-recommended method for transferring data between ARCHER2 and JASMIN is using GridFTP using certificate authentication. This allows data transfers to run on the ARCHER2 serial nodes using certificate-based authentication rather than SSH.... I think my credentials are all OK on the JASMIN (xfer1&2) and on archer2. But the transfer fails with output * * * [WARN] [SUBPROCESS]: Command: globus-url-copy -vb -cd -cc 4 -sync file:///work/n02/n02/schn02/archive/u-dc258/17730101T0000Z/ sshftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z/ [SUBPROCESS]: Error = 1: error: Unable to check destination url for sync: sshftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z/ an end-of-file was reached globus\_xio: An end of file occurred * * * I did a module load: module load gct/v6.2.20201212 and tried the command from a archer2 node and the same problem occurred. Is this still the recommended method for pptransfer? And if so any advice as the what the problem may be will be very gratefully received. Many thank, Andrew ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [19 January 2024 18:04 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/2 "2024-01-19T18:04:18Z")
Andrew Have you got access to [hpxfer1.jasmin.ac.uk](http://hpxfer1.jasmin.ac.uk) (high-performance transfer)? Grenville ---
**Author:** ![aschurer](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@aschurer](https://cms-helpdesk.ncas.ac.uk/u/aschurer)\ **Post date:** [22 January 2024 10:26 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/3 "2024-01-22T10:26:00Z")
Hi Grenville, Yes I have and can ssh from ARCHER2 to hpxfer1. And as far as I can understand have the correct credential set up at both ends (but do not totally understand the process so may have not done something quite right). e.g. schn02@ln02:~\> openssl x509 -in cred.jasmin -noout -startdate -enddate notBefore=Jan 19 15:24:34 2024 GMT notAfter=Feb 18 15:24:34 2024 GMT schn02@ln02:~\> ssh [aschurer@hpxfer1.jasmin.ac.uk](mailto:aschurer@hpxfer1.jasmin.ac.uk) [aschurer@hpxfer1 ~]$ openssl x509 -in cred.jasmin -noout -startdate -enddate notBefore=Jan 19 15:26:15 2024 GMT notAfter=Feb 18 15:26:15 2024 GMT ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [22 January 2024 11:18 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/4 "2024-01-22T11:18:50Z")
cred.jasmin needs to be in the ARCHER2 /work space. (It’s not needed on JASMIN) ---
**Author:** ![aschurer](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@aschurer](https://cms-helpdesk.ncas.ac.uk/u/aschurer)\ **Post date:** [22 January 2024 11:52 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/5 "2024-01-22T11:52:03Z")
I now have a cred.jasmin file on ARCHER2 in /work/n02/n02/schn02/ I originally had it in my home folder. So removed everything and started again in my work folder. schn02@ln02:/work/n02/n02/schn02\> openssl x509 -in cred.jasmin -noout -startdate -enddate notBefore=Jan 22 11:44:54 2024 GMT notAfter=Feb 21 11:44:54 2024 GMT But the transfer is still returning me an error. schn02@ln02:/work/n02/n02/schn02\> globus-url-copy -vb -cd -cc 4 -sync file:///work/n02/n02/schn02/archive/u-dc258/17730101T0000Z/ sshftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z error: Unable to check destination url for sync: sshftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z an end-of-file was reached globus\_xio: An end of file occurred Should this command be working? Is there any other way of testing my setup? Thanks again Andrew ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [22 January 2024 16:37 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/6 "2024-01-22T16:37:37Z")
Andrew Try deleting the `.globus` directory on JASMIN (or move it elsewhere) Grenville ---
**Author:** ![aschurer](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@aschurer](https://cms-helpdesk.ncas.ac.uk/u/aschurer)\ **Post date:** [22 January 2024 16:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/7 "2024-01-22T16:58:06Z")
I’ve now moved the directory as suggested but unfortunately still get the same problem. ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [22 January 2024 17:52 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/8 "2024-01-22T17:52:46Z")
Andrew, is this the full command you’re running? `schn02@ln02:/work/n02/n02/schn02> globus-url-copy -vb -cd -cc 4 -sync file:///work/n02/n02/schn02/archive/u-dc258/17730101T0000Z/ sshftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z` if so, it’s missing `-cred /work/n02/n02//cred.jasmin` ---
**Author:** ![aschurer](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@aschurer](https://cms-helpdesk.ncas.ac.uk/u/aschurer)\ **Post date:** [23 January 2024 12:12 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/9 "2024-01-23T12:12:23Z")
I was indeed missing the credential part. I’ve added this and my command is now: schn02@ln04:/work/n02/n02/schn02\> globus-url-copy -cred cred.jasmin -vb -cd -cc 4 -sync file:///work/n02/n02/schn02/archive/u-dc258/17730101T0000Z/ sshftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z/ It still produces the same error though error: Unable to check destination url for sync: sshftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z/ an end-of-file was reached globus\_xio: An end of file occurred Looking through the documentation, I found that gsiftp was often used so tried the following which worked. I don’t understand enough about this transfer to know the significance of this though… schn02@ln04:/work/n02/n02/schn02\> globus-url-copy -cred cred.jasmin -vb -cd -cc 4 -sync file:///work/n02/n02/schn02/archive/u-dc258/17730101T0000Z/ gsiftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z/ Source: file:///work/n02/n02/schn02/archive/u-dc258/17730101T0000Z/ Dest: gsiftp://gridftp1.jasmin.ac.uk/gws/nopw/j04/glosat/production/UKESM/raw/u-dc258/17730101T0000Z/ Thanks again, Andrew ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [26 January 2024 08:27 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/10 "2024-01-26T08:27:49Z")
Hi Andrew, You need to update version of the postproc branch in your suite. The one you are using doesn’t support gridftp with certificates, hence the incorrect gridftp command. In fcm\_make\_pp → configuration update the versions of the pp\_sources to: postproc\_2.3\_pptransfer\_gridftp\_nopw@4557 postproc\_2.3\_archer2@4988 Regards, Ros. ---
**Author:** ![aschurer](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@aschurer](https://cms-helpdesk.ncas.ac.uk/u/aschurer)\ **Post date:** [26 January 2024 09:24 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/11 "2024-01-26T09:24:50Z")
Hi Ros, Thanks for this. I’ve updated the branch as suggested. My model simulation is currently running using the old pptransfer (not gridftp) but the next time it stops I will change to gridftp, reload and try this. Thanks and kind regards Andrew ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [26 January 2024 16:43 UTC](https://cms-helpdesk.ncas.ac.uk/t/problem-with-pptransfer-from-archer2-jasmin/1308/12 "2024-01-26T16:43:20Z")
Hi Andrew, If you’re switching mid-run you will need to re-insert the `fcm_make_pp` and `fcm_make2_pp` tasks into the suite to re-build the postproc scripts. A reload alone won’t cause a rebuild of the scripts. After you have restarted/reloaded try: ```auto cylc insert --no-check u-dc258 fcm_make_pp. ``` That should insert the task, you may need to manually trigger it to run. Once that has completed do the same for the `fcm_make2_pp` task. ```auto cylc insert --no-check u-dc258 fcm_make2_pp. ``` Regards, Ros.