Update Globus transdfer for UKESM1.1-ice

1

Hello,

I’m updating my suite (u-dh564) so it can use Globus service. I’m following the instructions IMPORTANT: Retirement of JASMIN Gridftp server. The suite uses a special version of postproc with configuration:

  meta=archive_and_meaning/fcm_make/postproc_2.3
  
  [env]
  config_base=fcm:moci.xm-br/dev/robinsmith/postproc_2.3_unicicles_support
  config_rev=
  extract=extract
  install=build
  install_host=remote.cfg
  model_config=um-nemocice-unicicles.cfg
  pp_rev=postproc_2.3
  pp_sources=fcm:moci.xm-br/dev/rosalynhatcher/postproc_2.3_pptransfer_gridftp_nopw@3202 fcm:moci.xm-br/dev/robinsmith/postproc_2.3_unicicles_support
  verify_config=verify.cfg

It’s not the same as the ones from the instructions. Could you advice me how to set up the configuration for the Globus please?

All the best,
Jing

2

Hi Jingjin,

  • Please replace the revision number @3202 of the postproc_2.3_pptransfer_gridftp_nopw branch with @5411.

  • In app/postproc/rose-app.conf add the following variables to the [namelist:pptransfer] section and set gridftp=false:

    globus_cli=true
globus_default_colls=true
globus_notify='off'

If you are updating an already running suite then continue with step 3 of the instructions.

Regards,
Ros

3

Hi Ros,

Thanks for the instructions.

I’ve completed step 3, but pptransfer failed with error:

[WARN]  [SUBPROCESS]: Command: rsync -av --stats --rsync-path=mkdir -p /gws/nopw/j04/terrafirma/jjin/archer2/u-dh564/19240101T0000Z && rsync /work/n02/n02/jingjin562/cylc-run/u-dh564/share/cycle/19240101T0000Z/ gridftp1.jasmin.ac.uk:/gws/nopw/j04/terrafirma/jjin/archer2/u-dh564/19240101T0000Z
[SUBPROCESS]: Error = 255:
        ssh: connect to host gridftp1.jasmin.ac.uk port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(228) [sender=3.2.3]

[WARN]  Transfer command failed: rsync -av --stats --rsync-path="mkdir -p /gws/nopw/j04/terrafirma/jjin/archer2/u-dh564/19240101T0000Z && rsync" /work/n02/n02/jingjin562/cylc-run/u-dh564/share/cycle/19240101T0000Z/ gridftp1.jasmin.ac.uk:/gws/nopw/j04/terrafirma/jjin/archer2/u-dh564/19240101T0000Z
[ERROR]  transfer.py: Unknown Error - Return Code=255
[FAIL]  Command Terminated
[FAIL] Terminating PostProc...
[FAIL] transfer.py <<'__STDIN__'
[FAIL]
[FAIL] '__STDIN__' # return-code=1
2024-12-10T14:00:52Z CRITICAL - failed/EXIT

I’ve set gridftp false. I’m confused why they’re trying to connect gridftp server. Should I leave the remote host blank or do I miss some steps?

All the best,
Jing

4

Hi Jing,

You are updating a running suite so need to follow the instructions in step 3 to rebuild the postproc code. Ie. re-run the fcm_make_pp & fcm_make2_pp tasks.

Regards,
Ros

5

Hi Ros,

This error occurs after re-running the fcm_make_pp & fcm_make2_pp tasks. But I’ll do the re-run once again and see if it works.

All the best,
Jing

6

Hi Jing,

I’ve just taken a look at your suite and the settings you post above aren’t the ones being used by the suite. This suite has an optional configuration override for pp_sources which you can see in the rose edit GUI - indicated by the finger on a piece of paper icon.

This suite actually uses the following postproc branches:

pp_sources=fcm:moci.xm-br/dev/rosalynhatcher/postproc_2.3_terrafirma fcm:moci.xm_br/dev/annetteosprey/postproc_2.3_archer2@4988

which again are non-standard branches. I am currently in the process of modifying the postproc_2.3_terrafirma branch for globus. Once this is done, hopefully tomorrow, I will let you know to re-try.

Regards,
Ros

7

Hi Jing,

I have now updated the postproc_2.3_terrafirma branch please now go ahead and re-run fcm_make_pp and fcm_make2_pp tasks.

Regards,
Ros.

8

Hi Ros,

Many thanks for your help. The pptransfer task is working Okay. The files appear to be in my JASMIN folder.

All the best,
Jing

9

Great. :+1: Thanks for letting me know.

Cheers,
Ros