Hi Charlie,
I’ve emailed Jeremy to see if he has that directory on Monsoon3 (that’s a monsoon2 path). I can’t see into his home directory to check. If he hasn’t then you’ll need to check it out from the moci MOSRS repository yourself.
Cheers,
Ros
Thanks very much Ros.
Firstly, can you remind me how I do that i.e. check out the moci repository?
Secondly, despite the meta data not being quite right, I tried running the workflow anyway (using cylc vip). The good news is that it did at least begin running. The bad news is that it seems to have failed at the fcm_make2_ocean task (fcm_make_ocean seems to have succeeded). Meaning it didn’t even get to the nemo_cice task, which is the one I really care about. But I don’t know how, using the new system, to check where the error lies.
On that note: is there any documentation for the new system, over and above what is at Summary Of Major Changes — Cylc 8.6.3 documentation ? Because, dare I say it, it (and the Cylc 8 Migration Guide in general) is not particularly user friendly, at least not for a dummy like me. It assumes a lot of background knowledge. Looking at the new GUI (see below), I can see that purple now = submit failed and that green = succeeded. But where are the various error and log files? And I also note that the structure of the output has changed, such that there is now a run1 and run2. What do these mean? Intuitively, are these literally the number of times I have run this workflow e.g. run1 was my attempt several weeks ago before I had upgraded, and run2 is what I did yesterday? Certainly ~/cylc-run/u-dg710/run1 appears to contain empty directories. Likewise, ~/cylc-run/u-dg710/share and /work now appears to be ~/cylc-run/u-dg710/run2/share and /work. Is that correct or am I looking in the wrong place?
Sorry for the questions.
Charlie
Hi Charlie,
Jeremy is currently out of the office and I’ve had a quick look in the moci repos and it isn’t his branch so will need to do a bit more investigation to track it down or wait for Jeremy to come back. Not having the metadata doesn’t affect the running of the job in any way, just makes the GUI not so nice.
You can see what all the coloured and grey blobs mean here:
In the cylc GUI if you click on the purple square a menu will pop up with an option to take you to the log files within the GUI.
Alternatively you can find then on PUMA2/ARCHER2 under ~/cylc-run/<suiteid>/<runX>/logs
More details here:
Yes with cylc 8 everytime you run the workflow with cylc vip <id> it will start a completely new run in a new run[1,2,…] directory.
Cheers,
Ros
Okay, thank you very much for the extra information.
I have now checked the log files, and it would appear (as I suspected) that the fcm_make2_ocean task has failed at the submission stage:
2026-05-18T14:40:43Z [STDERR] qsub: error: [PBSSitePolicy] export of complete environment with -V is no longer supported
[((‘event-handler-00’, ‘submission failed’), 1) cmd] rose suite-hook ‘submission failed’ ‘u-dg710/run2’ ‘19760101T0000Z/fcm_make2_ocean’ ‘job submission failed’
[((‘event-handler-00’, ‘submission failed’), 1) ret_code] 1
[((‘event-handler-00’, ‘submission failed’), 1) err] Command obsolete, use Cylc event handlers
So I’m guessing this is almost certainly something I did wrong in my new flow.cylc? It has to be, because this workflow ran perfectly well back in November, before we switched to Monsoon3.
Charlie
Hi Charlie,
In [[HPC_UM]] remove the -V =. This option to PBS is not supported on Monsoon3.
Cheers,
Ros
Hi Charlie,
I’ve also found the metadata branch. You can check it out from the moci repository with:
fcm co fcm:moci.x-br/dev/ericaneininger/r177_metatdata_coupled_apps
Then edit app/nemo_cice/rose-app.conf to point your suite at it.
Cheers,
Ros
Sorry Ros, I have tried doing that, i.e. I changed the first line in my ~/roses/u-dg710/app/nemo_cice/rose-app.conf to
meta=/home/users/charles.williams.ext/r177_metatdata_coupled_apps
(I wasn’t sure if it needed to be checked out to any particular place, but my home directory seemed as good as any), but I get the same error - see below. Worse, now the rose config-edit won’t even open the GUI, it just hangs with the below. I definitely saved the file, so why is it still looking for Jeremy’s path?
Charlie
Further to this… Sorry, the GUI does open, but only when I close down the error box. And the meta data is still not right, once it is open (unsurprisingly, as it is still looking for the wrong version rather than mine).
Charlie
And sorry, I realise we are dealing with 2 separate threads here, but in the meantime I tried running it again after making that change (i.e. removing -V), but same error message as before.
[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=8 -oStrictHostKeyChecking=no login12 env CYLC_VERSION=8.6.3 CYLC_ENV_NAME=cylc-8.6.3-2 bash --login -c ‘’“'”‘exec “$0” “$@”’“'”‘’ cylc jobs-submit --utc-mode --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin – ‘$HOME/cylc-run/u-dg710/run1/log/job’ 19760101T0000Z/fcm_make2_ocean/01
[jobs-submit ret_code] 159
[jobs-submit out] 2026-05-20T15:07:38Z|19760101T0000Z/fcm_make2_ocean/01|159|None
2026-05-20T15:07:38Z [STDERR] qsub: Unauthorized Request
[((‘event-handler-00’, ‘submission failed’), 1) cmd] rose suite-hook ‘submission failed’ ‘u-dg710/run1’ ‘19760101T0000Z/fcm_make2_ocean’ ‘job submission failed’
[((‘event-handler-00’, ‘submission failed’), 1) ret_code] 1
[((‘event-handler-00’, ‘submission failed’), 1) err] Command obsolete, use Cylc event handlers
Given that both of these problems seem to be similar i.e. it is not picking up a change I have just made, have I missed something silly e.g. I need to install it or validate it again before running? So cylc install or cylc validate, before cylc vip?
Charlie
