Postproc fails on Monsoon

Rose/Cylc and FCM
1

Hello,

I am running GOSI9 simulations on Monsoon and everything has been mostly fine i.e. now at year 24 of model spin up. One of the simulations has failed on postproc and I’m unsure how to fix it. I’d rather try and fix it than go back nearly a year in the simulation process from a restart file.

Here is some of the output from the job.err file

[WARN]  [SUBPROCESS]: Command: moo put -f -vv /home/d02/twilder/cylc-run/u-db080/share/data/History_Data/NEMOhist/db080o_19981201_restart.nc moose:crum/u-db080/oda.file
[SUBPROCESS]: Error = 2:
	put command-id=1853343882 failed: (SSC_TASK_REJECTION) one or more tasks are rejected.
  /home/d02/twilder/cylc-run/u-db080/share/data/History_Data/NEMOhist/db080o_19981201_restart.nc -> moose:/crum/u-db080/oda.file/db080o_19981201_restart.nc: (TSSC_CONFLICT_WITH_EARLIER_COMMAND) command conflicts with another command.
  Conflicting command-ids: 1853315084, 
put: failed (2)

[WARN]  moo.py: Moose Error: user-error (see Moose docs). (ReturnCode=2) File: /home/d02/twilder/cylc-run/u-db080/share/data/History_Data/NEMOhist/db080o_19981201_restart.nc
[WARN]  Failed to archive file: db080o_19981201_restart.nc. Will try again later.
[FAIL]  main_pp.py - PostProc complete. Exiting with errors in nemo_archive
[FAIL] Terminating PostProc...
[FAIL] main_pp.py nemo cice # return-code=1
2024-06-04T15:47:49Z CRITICAL - failed/EXIT

Any idea how this can be fixed?

Cheers

2

Are you still having the problem? - I note:

At 10:00 BST on Tuesday 4th June there will be an outage to MOOSE/MASS in order to shut down the Production database in order to apply the latest Oracle patch. This is scheduled between 10:00 BST and 12:00 BST but may end sooner. All MOOSE 'PUT, 'GET', 'SELECT' and 'FILTER' commands will fail.

but the timing doesn’t match with the failures in u-db080

Grenville

3

It has now run successfully. Very strange. None of my other runs had any mass issues, but it must be related.