Warm start on PUMATest not working the 1st two times

Unified Model
, ,
1

Hi CMS Helpdesk:
I just tried a warm start on PUMATest, after the PUMA failure or whatever the name of the failure is.
The first time I did it, it complained about Stale file handles. It didn’t look too serious, so I just tried a warm start a 2nd time, wherein it says ‘Cannot call rmtree on a symbolic link’. I tried a 3rd time, and it seems to be running. See the first two times console log below.
I am just alerting you, in case the first two warm starts shouldn’t have happened that way or something.
Patrick McGuire

-bash-4.1$ rose suite-run -- --warm 20030601T0000Z
[INFO] export CYLC_VERSION=7.8.7
[INFO] export ROSE_ORIG_HOST=pumatest.nerc.ac.uk
[INFO] export ROSE_SITE=
[INFO] export ROSE_VERSION=2019.01.3
[INFO] create: log.20220217T100757Z
[INFO] delete: log
[INFO] symlink: log.20220217T100757Z <= log
[INFO] log.20220217T100731Z.tar.gz <= log.20220217T100731Z
[INFO] delete: log.20220217T100731Z/
[INFO] create: log/suite
[INFO] create: log/rose-conf
[INFO] symlink: rose-conf/20220217T100757-run.conf <= log/rose-suite-run.conf
[INFO] symlink: rose-conf/20220217T100757-run.version <= log/rose-suite-run.version
[INFO] install: app
[INFO]     source: /home/pmcguire/roses/u-cl528/app
[INFO] REGISTERED u-cl528 -> /home/pmcguire/cylc-run/u-cl528
[FAIL] ssh -oBatchMode=yes -n login3.archer2.ac.uk env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.7\ bash\ -l\ -c\ \'\"$0\"\ \"$@\"\'\ rose\ suite-run\ -vv\ -n\ u-cl528\ --run=run\ --remote=uuid=d2e24ea8-897a-4d9f-a5c1-6bf56a0ce2c0,now-str=20220217T100757Z,root-dir=\'$DATADIR\' # return-code=2, stderr=
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar -czf log.20220203T101723Z.tar.gz log.20220203T101723Z # return-code=2, stderr=
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19820601T0000Z/atmos_main/01/job.out: file changed as we read it
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19820601T0000Z/atmos_main/01/job.status: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19820601T0000Z/atmos_main/01/job.err: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19820601T0000Z/atmos_main/01: file changed as we read it
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19820601T0000Z/atmos_main: file changed as we read it
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19820601T0000Z/postproc: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19820601T0000Z: file changed as we read it
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19711201T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19760601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19640601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19610601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19910601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19940601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19921201T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19531201T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/20011201T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19970601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19490601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/20030601T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job/19621201T0000Z: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/job: file changed as we read it
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z/suite: Cannot stat: Stale file handle
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: log.20220203T101723Z: file changed as we read it
[FAIL] [FAIL] 2022-02-17T10:09:32+0000 tar: Exiting with failure status due to previous errors
-bash-4.1$ rose suite-run -- --warm 20030601T0000Z
[INFO] export CYLC_VERSION=7.8.7
[INFO] export ROSE_ORIG_HOST=pumatest.nerc.ac.uk
[INFO] export ROSE_SITE=
[INFO] export ROSE_VERSION=2019.01.3
[INFO] create: log.20220217T101042Z
[INFO] delete: log
[INFO] symlink: log.20220217T101042Z <= log
[INFO] log.20220217T100757Z.tar.gz <= log.20220217T100757Z
[INFO] delete: log.20220217T100757Z/
[INFO] create: log/suite
[INFO] create: log/rose-conf
[INFO] symlink: rose-conf/20220217T101042-run.conf <= log/rose-suite-run.conf
[INFO] symlink: rose-conf/20220217T101042-run.version <= log/rose-suite-run.version
[INFO] REGISTERED u-cl528 -> /home/pmcguire/cylc-run/u-cl528
[FAIL] ssh -oBatchMode=yes -n login.archer2.ac.uk env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.7\ bash\ -l\ -c\ \'\"$0\"\ \"$@\"\'\ rose\ suite-run\ -vv\ -n\ u-cl528\ --run=run\ --remote=uuid=f05aa579-0431-4c84-9f58-7be23e33cc73,now-str=20220217T101042Z,root-dir=\'$DATADIR\' # return-code=1, stderr=
[FAIL] [FAIL] 2022-02-17T10:10:45+0000 Cannot call rmtree on a symbolic link
2

Thanks Patrick.

I’ll add these to the webpage in case others encounter this too. I saw the stale file handle on one of my suites when I was testing but never saw it again.

Cheers,
Ros.

3

You’re welcome, Ros!
I also saw similar messages (but not the same) the 1st two times I warm started a 2nd suite u-cl527.
But the 2nd suite also was able to be warm started on the 3rd attempt on PUMATest.
Patrick