Cylc submission failures

1

I was running a bunch of suites – all automatically submitted. I have 10 suites running at once which should be within my limits. The Q size is limited to 2 jobs and I have 11 other serial jobs in the Q. All suites failed in atmos_main and in postproc with (from a sample) failures of the form:

[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=8 -oStrictHostKeyChecking=no ln04 env CYLC_VERSION=8.6.2 CYLC_ENV_NAME=cylc-8.6.2-2 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-submit --debug --utc-mode --remote-mode --clean-env --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/opt_dfols46/dn00j/log/job' 20110301T0000Z/atmos_main/07
[jobs-submit ret_code] 1
[jobs-submit out] 2026-01-23T10:33:08Z|20110301T0000Z/atmos_main/07|1|None
2026-01-23T10:33:08Z [STDERR] sbatch: error: AssocMaxCpuMinutesPerJobLimit
2026-01-23T10:33:08Z [STDERR] sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

This is the 2nd (for postproc) and 3rd cycle (for atmos_main) so I don’t think it is inherent to having 10 suites running at once. And the system ran 10 suites to completion earlier.

atmos_main looks like to have reattempted 7 times so far…

So, what is happening? Have I ran out of time…

Simon

2

Hi Simon,

The error message is a bit misleading here. The issue is that the project code you are using n02-TERRAFIRMA had run out of resource. I have topped it up so it should now work.

Annette

3

Hi Annette,

thanks! That seems to have solved my problems as I triggered all the failed jobs and they have all been submitted…

Simon

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.