Almost all of the coupled jobs I am trying to run are failing with a time limit reached error - e.g. JOB 15754614 ON nid001458 CANCELLED AT 2026-10-27T03:24:39 DUE TO TIME LIMIT. While digging into it to try understand what’s going wrong, I find that the time.step for the ocean reaches the expected point and as far as I can tell all the output files that I expect to be in the work directory are there. For an example, see /work/n02/n02/ambwalsh/cylc-run/u-dy890/run64/work/20150101T0000Z/coupled/
Note that the output is written out by 05:17 and the job officially fails at 07:22.
I have two questions about this. Firstly, is there a known possible cause for these time outs (e.g. I have had issues with unrelated simulations and the latest SLURM update on ARCHER2). Secondly, since the simulation seems to be reaching the end, even if the coupled job isn’t exiting cleanly, can I force Cylc to continue the run? I vaguely recall being able to set the state of a job to succeeded and then hopefully the subsequent jobs will start running with no issue.
I’m not running on ARCHER2 at the moment but I believe there was an issue with the file system reported yesterday. I’ll see if I can find out more about this…
The Cylc command you’ll want to use is cylc set [options]. You can run this in the terminal, in the web UI (click on the task and navigate the menu), or use the terminal user interface (TUI).
$ cylc set -h|head
Usage: cylc set [OPTIONS] ID ...
Manually complete task outputs, and satisfy task and xtrigger prerequisites.
By default this command completes required outputs, plus the "submitted",
"started", and "succeeded" outputs even if they are optional.
Setting prerequisites promotes target tasks to the n=0 active window; setting
outputs also sets the prerequisites of any tasks that depend on those outputs.
Thanks for your help! I have begun using cylc set as a workaround which is working well.
I also heard back from the ARCHER2 service desk and they confirmed there are issues with high I/O workload due to people copying data off the machine. Not sure how that directly leads to this issue, but it unfortunately seems like it may not be fixable.
Great news that that fix is working in the meantime.
I can’t say exactly why the file system would stop reporting task states back but it does make sense in the sense that ARCHER2 needs to communicate with PUMA2 in order to ‘poll’ tasks states which are then reported in the cylc web UI. I’m actually seeing similar behaviour myself right now with a Cylc workflow on ARCHER2 that I am about to port to the new Cirrus machine.