Issue with globus transfer on archer

1

I have been trying to work out all morning why this pp transfer failed, I attach the job err from the u-cm655_da191_20181229-W_naew-11 run. I don’t understand why there would be a conflict with a transfer with identical paths not yet completed, and what I should do?

[WARN] file:nemocicepp.nl: skip missing optional source: namelist:moose_arch

[WARN] file:atmospp.nl: skip missing optional source: namelist:script_arch

[WARN] file:nemocicepp.nl: skip missing optional source: namelist:script_arch

[WARN] [SUBPROCESS]: Command: globus transfer --format unix --jmespath task_id --recursive --fail-on-quota-errors --sync-level checksum --label u-cm655_da191_20081229-W_naew-11/20081201T0000Z --verify-checksum --notify off 3e90d018-0d05-461a-bbaf-aab605283d21:/work/n02/n02/egentile/cylc-run/u-cm655_da191_20081229-W_naew-11/share/cycle/20081201T0000Z a2f53b7f-1b4e-4dce-9b7c-349ae760fee0:/gws/nopw/j04/canari/users/bjharvey/archive/u-cm655_da191_20081229-W_naew-11/20081201T0000Z

[SUBPROCESS]: Error = 1:

Globus CLI Error: A Transfer API Error Occurred.

HTTP status: 409

request_id: uMmE8iLR4

code: Conflict

message: A transfer with identical paths has not yet completed

[WARN] Transfer command failed: globus transfer --format unix --jmespath ‘task_id’ --recursive --fail-on-quota-errors --sync-level checksum --label u-cm655_da191_20081229-W_naew-11/20081201T0000Z --verify-checksum --notify off 3e90d018-0d05-461a-bbaf-aab605283d21:/work/n02/n02/egentile/cylc-run/u-cm655_da191_20081229-W_naew-11/share/cycle/20081201T0000Z a2f53b7f-1b4e-4dce-9b7c-349ae760fee0:/gws/nopw/j04/canari/users/bjharvey/archive/u-cm655_da191_20081229-W_naew-11/20081201T0000Z

[ERROR] transfer.py: Globus Error: Network or server error occurred (Globus ReturnCode=1)

[FAIL] Command Terminated

[FAIL] Terminating PostProc…

[FAIL] transfer.py <<‘STDIN’

[FAIL]

[FAIL] ‘STDIN’ # return-code=1

2025-08-16T01:32:37Z CRITICAL - failed/EXIT

2

Hi Emanuele,

It looks like the transfer failed because another Globus task for the exact same source/destination paths is already running or retrying.

You don’t need to resubmit yet. Run globus task list and check for an ACTIVE task with the same label/paths. If it’s still retrying, just wait for it to finish.

If you want to restart cleanly, cancel it with globus task cancel and put your task id and then rerun the transfer once.

Best regards,

Juan

3

Hi Juan,

I terminated it from the web interface and then retriggered it from the cylc-rose suite.

But now I get a different error with globus about connection error

Bests,

Emanuele

Lmod is automatically replacing “cce/15.0.0” with “gcc/11.2.0”.

Due to MODULEPATH changes, the following have been reloaded:

  1. cray-mpich/8.1.23

[WARN] file:atmospp.nl: skip missing optional source: namelist:moose_arch

[WARN] file:nemocicepp.nl: skip missing optional source: namelist:moose_arch

[WARN] file:atmospp.nl: skip missing optional source: namelist:script_arch

[WARN] file:nemocicepp.nl: skip missing optional source: namelist:script_arch

[WARN] [SUBPROCESS]: Command: globus transfer --format unix --jmespath task_id --recursive --fail-on-quota-errors --sync-level checksum --label u-cm655_da191_20081229-W_naew-11/20081201T0000Z --verify-checksum --notify off 3e90d018-0d05-461a-bbaf-aab605283d21:/work/n02/n02/egentile/cylc-run/u-cm655_da191_20081229-W_naew-11/share/cycle/20081201T0000Z a2f53b7f-1b4e-4dce-9b7c-349ae760fee0:/gws/nopw/j04/canari/users/bjharvey/archive/u-cm655_da191_20081229-W_naew-11/20081201T0000Z

[SUBPROCESS]: Error = 1:

Globus CLI Error: A Globus Error Occurred.

error_type: GlobusConnectionError

message: ConnectionError on request

4

Hi Emanuele,

This is a GlobusConnectionError (the CLI cannot reach Globus or a collection is offline).

Can you try to check and reactivate the conectivity.

globus collection show 3e90d018-0d05-461a-bbaf-aab605283d21
globus collection show a2f53b7f-1b4e-4dce-9b7c-349ae760fee0
globus collection auto-activate 3e90d018-0d05-461a-bbaf-aab605283d21
globus collection auto-activate a2f53b7f-1b4e-4dce-9b7c-349ae760fee0

If it still fails, please re-run with verbose -v and send the last 30 lines

Best regards,

Juan

5

Also please on archer2 check type these commands to get more information:

ARCHER2> module load globus-cli/3.35.2 

ARCHER2> globus ls 3e90d018-0d05-461a-bbaf-aab605283d21:/~/ 

ARCHER2> globus ls a2f53b7f-1b4e-4dce-9b7c-349ae760fee0:/~/

Im sharing also further instructions here:

https://cms.ncas.ac.uk/unified-model/pptransfer-globus/

Juan

6

Hi Juan,

This is what globus collection tells me

egentile@ln01:/work/n02/n02/egentile/cylc-run/u-cm655_da191_20081229-W_naew-11/share/cycle> globus collection show 3e90d018-0d05-461a-bbaf-aab605283d21

MissingLoginError: Missing ‘manage_collections’ consent on an endpoint.

Please run:

globus login --gcs 01eaceb2-fdab-4509-8b6c-9ba112ab5f18

egentile@ln01:/work/n02/n02/egentile/cylc-run/u-cm655_da191_20081229-W_naew-11/share/cycle> globus collection show a2f53b7f-1b4e-4dce-9b7c-349ae760fee0

MissingLoginError: Missing ‘manage_collections’ consent on an endpoint.

Please run:

globus login --gcs 26ee7546-0b70-42b5-8f8d-6e443323ea82

7

Hi Juan,

Thanks a lot for your kind help.

8

Hi Emanuele,

Could you please try point 3 & 5 on this page: https://cms.ncas.ac.uk/unified-model/pptransfer-globus/ to authenticate to globus. Then check if everything runs ok. This should fix the manage_collections consent problem.

Best regards,

Juan

9

Hi Juan

Thanks it got fixed now!

Emanuele

10

Hi Emanuele,

Good that it got solved now!

Best regards,

Juan