I had ten simulations fail at 04:07 (ish) today with errors of the form:
sys-11 : UNRECOVERABLE error on system request
Resource temporarily unavailable
Encountered during a sequential formatted WRITE to unit 103
Fortran unit 103 is connected to a sequential formatted text file:
"pe_output/dn00m.fort6.pe.stdout"
Current format: (A)
^
sys-122 : FATAL error closing unit 103 during program termination
Had archer2 disk space filled up again?…Though data is going to the a2fs-nvme space so that seems unlikely.
I removed (cylc clean) some old simulations and resubmitted one, waited a while and then resubmitted the rest. All (right now) are running.
Simon