Max number of archer jobs exceeded

[shakka]

Hi CMS,

Is there a way to increase the number of allowable jobs I can run on archer2?

I am currently running ten years (in ten different suites) at once, but getting errors related to the number of jobs I’m allowed to run. I’m guessing this is an archer queue thing?

Cheers
Ella

[dcase]

You should be able to run 16 jobs concomitantly on the standard queue (and queue 64):
https://docs.archer2.ac.uk/user-guide/scheduler/

but any error message may have more detailed information.

Coordinating things across different suites is hard - if you run ensembles from the same suite then cylc has a few things you could do.

[shakka]

Thanks Dave. I guess it’s because I have multiple cycles running across all the nesting suites I’m trying to run… I assumed that this limit would be hard-wired but just thought I’d check in case it could be amended. As it’s not an ensemble I’m not sure how helpful the cylc option will be. Cheers for the speedy reply! E

[shakka]

Okay, the plot thickens… Now none of my jobs are running (some submitted, some waiting, but none running) and lots are submit-retrying or submit-failed with an error like this:

(ln03) 2023-11-08T13:32:02Z [STDERR] sbatch: error: QOSMaxSubmitJobPerUserLimit
(ln03) 2023-11-08T13:32:02Z [STDERR] sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user’s size and/or time limits)

Any idea what’s going on?

[dcase]

I don’t know what’s going on at A2, but I’d probably let it work itself out. Hopefully things are less costive soon.