# Unexpected timeout **URL:** \ **Category:** Unified Model\ **Tags:** ARCHER2\ **Created:** [13 October 2026 10:45 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195 "2026-10-13T10:45:03Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![c.j.r.williams](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/c.j.r.williams/32/591_2.png) [@c.j.r.williams](https://cms-helpdesk.ncas.ac.uk/u/c.j.r.williams)\ **Post date:** [13 October 2026 10:45 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/1 "2026-10-13T10:45:03Z")
Hi, Sorry to bother you, but my previously-successful suite (u-ed255) has failed giving me the following error message: > slurmstepd: error: \*\*\* STEP 15604805.0+0 ON nid003631 CANCELLED AT 2026-10-13T09:30:31 DUE TO TIME LIMIT \*\*\* which you can see at ~/cylc-run/u-ed255/log/job/19340101T0000Z/coupled/NN/job.err There is no other obvious error. I have seen this before, and it usually means that the coupled task has timed out. So I have increased the wallclock time, from 8 hours to 10 hours, then I did a `rose suite-run --reload`. What I don’t understand is why. This suite has run successfully (with the occasional instability, requiring a restart, but otherwise successfully) for over 80 cycles i.e. 80 years. All with a wallclock time of 8 hours. Why, if it completed all of these cycles within 8 hours, would it suddenly timeout now? Looking at my output, it got to November, therefore almost at the end of the year, before failing. So why would this cycle (which ran overnight last night, beginning just after midnight) suddenly take longer than all of the others? Maybe some sort of machine blip that happened overnight, i.e. nothing actually to do with my run? Thank you, Charlie ---
**Author:** ![c.j.r.williams](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/c.j.r.williams/32/591_2.png) [@c.j.r.williams](https://cms-helpdesk.ncas.ac.uk/u/c.j.r.williams)\ **Post date:** [15 October 2026 09:23 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/2 "2026-10-15T09:23:07Z")
Hi again, Further to this, another of my suites also failed yesterday for the same reason, i.e. the coupled task timed out. Again, I increased the wall clock time and it appears to be okay. But at the same time, yet another suite failed at the building stage (specifically fcm\_make2\_um), which also timed out. It is currently set to 30 minutes, but please can you remind me how increase this? Is it in my suite.rc? Either way, something has most definitely changed in the last ~48 hours, in that everything seems to be running slower. As I said, my 8 hour wall clock time for the coupled task has been absolutely fine for the last 80 cycles, yet now it isn’t. Likewise my 30 minute wall clock time for the fcm\_make2\_um task has been absolutely fine, yet now it isn’t. I can see other people here have reported ARCHER2 slowness, is this all related? I appreciate it is shutting down in just over 4 weeks, but I was really really counting on running my simulations until the very last minute. They wouldn’t be unofficially closing it down early, would they? Charlie ---
**Author:** ![AnnetteOsprey](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/annetteosprey/32/27_2.png) [@AnnetteOsprey](https://cms-helpdesk.ncas.ac.uk/u/AnnetteOsprey)\ **Post date:** [15 October 2026 09:39 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/3 "2026-10-15T09:39:35Z")
Hi Charlie, I have also noticed I/O slow down in my jobs. I think it is probably due to people transferring data off the service which is stressing the filesystem. You will just need to increase the runtime. I understand your frustration as many of us are trying to keep running for the last few weeks. Please do report these issues to the Archer2 helpdesk if you are concerned. Annette ---
**Author:** ![AnnetteOsprey](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/annetteosprey/32/27_2.png) [@AnnetteOsprey](https://cms-helpdesk.ncas.ac.uk/u/AnnetteOsprey)\ **Post date:** [15 October 2026 09:47 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/4 "2026-10-15T09:47:59Z")
Also where you add in the time limit for the UM build depends on the suite, but assuming it is set up the same as u-ed255, you would edit the `site/archer2.rc` file. Navigate to the `UMBUILD_RESOURCE` block, and add the time limit in there: ```auto [[UMBUILD_RESOURCE]] inherit = None, HPC_SERIAL [[[job]]] execution time limit = PT1H ``` Hope this helps, Annette ---
**Author:** ![c.j.r.williams](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/c.j.r.williams/32/591_2.png) [@c.j.r.williams](https://cms-helpdesk.ncas.ac.uk/u/c.j.r.williams)\ **Post date:** [15 October 2026 10:02 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/5 "2026-10-15T10:02:56Z")
Thank you very much Annette, brilliant and really helpful as always. I thought it might be in my site/archer2.rc, but I didn’t have that particular job line in my resource block. It was: ```auto [[UMBUILD_RESOURCE]] inherit = None, HPC_SERIAL [[[environment]]] ROSE_TASK_N_JOBS = 6 [[[directives]]] --ntasks=6 ``` so it now reads: ```auto [[UMBUILD_RESOURCE]] inherit = None, HPC_SERIAL [[[job]]] execution time limit = PT1H [[[environment]]] ROSE_TASK_N_JOBS = 6 [[[directives]]] --ntasks=6 ``` Is that correct? Charlie ---
**Author:** ![c.j.r.williams](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/c.j.r.williams/32/591_2.png) [@c.j.r.williams](https://cms-helpdesk.ncas.ac.uk/u/c.j.r.williams)\ **Post date:** [16 October 2026 14:22 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/6 "2026-10-16T14:22:42Z")
Hi Annette, Sorry to bother you again, as I get the feeling you are probably getting lots of messages along these lines. THE ARCHER2 slowdown now seems to be seriously impacting my simulations more than just requiring more wall clock time. See below for a screenshot of my current status. In short, the postprocessing and pptransfer tasks from 2 cycles ago are still in the queue, which means so are the previous cycle. And for my current cycle (although the coupled task has finished), the postprocessing and pptransfer tasks haven’t even appeared. Indeed, the next cycle hasn’t even appeared (it should be set to run for at least another 25 years). Presumably the next cycle hasn’t appeared because of the bottleneck with the previous cycles? In other words, everything has become completely snarled up. What you can’t see with the screenshot is that the submitted tasks from the 1935 cycle were actually submitted at 9 PM yesterday (Thursday), so they have been in the queue now for almost 24 hours. It also means, of course, that even if the next coupled task was to appear and run, I am very close to running out of space (because none of the previous pptransfer and housekeeping tasks have run). I am now at 5.5 T, out of quota of 6 T, on /work. As you suggested, I contacted the ARCHER2 helpdesk with my concerns. I wasn’t going to as I don’t expect them to do anything about it, but Ros persuaded me it was worth it. In the meantime, is there anything I can do manually in order to get my simulation going again? I believe I could quite easily do the transfer myself, manually, using Globus. And obviously I could do the deleting myself, once transferred. Which presumably means I could then set these tasks to succeeded, which would trigger the next cycle to appear? The only tasks I am unsure about doing manually are the various postprocessing tasks, as I’m never entirely sure exactly what these do. Am I right in thinking that the main job of these is to rebuild the various ocean files e.g. rebuilding the _n_ ocean restart files, one for each processor, into one? If that’s all these tasks do, then I can do it myself. But what about the other tasks e.g. postproc\_atmos? What does this do, and would it be terribly difficult to do it manually? Thanks a lot, Charlie PS. I am making an assumption here, that this bottleneck is indeed because of the slowdown. But I guess that assumption might be wrong, and there might be something else wrong with my suite? ![image](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/9bcfa1016f733394b04ae7e10e1690d5ebd9924d.png) ---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [16 October 2026 17:17 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/7 "2026-10-16T17:17:52Z")
Hi Charlie, The snarl up with the serial queue is due to the maintenance being carried out on the slurm scheduler database. See the ARCHER2 status page. The serial nodes were being drained over the last 24hours in order to apply the update, this has resulted in a huge backlog of serial jobs, which was exacerbated by a couple of longer running jobs holding up the queue further. The serial queues are running again now so hopefully things will start moving normally over the weekend. Cylc is set so tasks don’t run too many cycles ahead of stalled/stuck ones as this can cause problems e.g. running out of disk space, and if there is a failure somewhere it can make things a lot harder to rectify. I would not suggest trying to do the postproc tasks manually as it could very easily get in a pickle. Cheers, Ros ---
**Author:** ![c.j.r.williams](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/c.j.r.williams/32/591_2.png) [@c.j.r.williams](https://cms-helpdesk.ncas.ac.uk/u/c.j.r.williams)\ **Post date:** [16 October 2026 17:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/unexpected-timeout/2195/8 "2026-10-16T17:44:00Z")
Thanks very much, all understood about the maintenance, at least it’s something that should resolve. Yes, I thought that was why the cycles weren’t continuing. And don’t worry, I would much rather not try to do the postprocessing manually, as getting into a pickle would be almost guaranteed! Have a good weekend, Charlie