Hi,
I’m currently running 3 UKESM1.1 jobs on ARCHER2 which have been running successfully for a number of weeks.
u-cu100, u-cu101 & u-cu102
They have all failed during the “coupled” task.
They all ran for a bit of the task producing data and then fail with errors “CANCELLED DUE TO TIME LIMIT” (see below)
They have all failed this morning - but not at the same point in the model runs.
I can’t see the reason why they’ve failed so any help would be gratefully received.
Many thanks
Andrew
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 3086676.0+0 ON nid006471 CANCELLED AT 2023-02-14T13:29:18 DUE TO TIME LIMIT ***
slurmstepd: error: *** STEP 3086676.0+2 ON nid006631 CANCELLED AT 2023-02-14T13:29:18 DUE TO TIME LIMIT ***
slurmstepd: error: *** JOB 3086676 ON nid006471 CANCELLED AT 2023-02-14T13:29:18 DUE TO TIME LIMIT ***
slurmstepd: error: *** STEP 3086676.0+1 ON nid006630 CANCELLED AT 2023-02-14T13:29:18 DUE TO TIME LIMIT ***
srun: error: nid006631: tasks 0-5: Terminated
srun: launch/slurm: _step_signal: Terminating StepId=3086676.0+0
srun: launch/slurm: _step_signal: Terminating StepId=3086676.0+2
srun: launch/slurm: _step_signal: Terminating StepId=3086676.0+1