Failure from rose suite-run

Hi,
I am trying to run suite u-db898 (which is a couple of years old) use rose suite-run:
rose suite-run -v -C roses/u-db898

and get an error:
[FAIL] 2025-05-07T10:50:33+0100 ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n ln01 env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-db898\ –run=run\ –remote=uuid=01542424-c658-4f69-ae77-ef5f5adfcbd8,now-str=20250507T095027Z,root-dir='$DATADIR' # return-code=1, stderr=
[FAIL] 2025-05-07T10:50:33+0100 This command has been replaced by: “cylc install”.

What do I need to do to fix this? I fear it is something in my puma2 environment.
Simon

Full debug info:
tetts@puma2 ~]$ rose suite-run -v -C roses/u-db898
[INFO] chdir: roses/u-db898/
[INFO] Configuration: /home4/home/n02-puma/tetts/roses/u-db898/
[INFO] file: rose-suite.conf
[INFO] export CYLC_VERSION=7.8.12
[INFO] export ROSE_ORIG_HOST=puma2.archer2.ac.uk
[INFO] export ROSE_SITE=
[INFO] export ROSE_VERSION=2019.01.3
[INFO] create: log.20250507T094601Z
[INFO] delete: log
[INFO] symlink: log.20250507T094601Z <= log
[INFO] log.20250507T091115Z.tar.gz <= log.20250507T091115Z
[INFO] delete: log.20250507T091115Z/
[INFO] create: log/suite
[INFO] create: log/rose-conf
[INFO] symlink: rose-conf/20250507T104601-run.conf <= log/rose-suite-run.conf
[INFO] symlink: rose-conf/20250507T104601-run.version <= log/rose-suite-run.version
[INFO] unchanged: ana/mule_cumf.py
[INFO] source: svn://puma2.archer2.ac.uk/um.xm/main/trunk/rose-stem/ana/mule_cumf.py@71790 (fcm:um.xm_tr/rose-stem/ana/mule_cumf.py@vn11.4)
[INFO] unchanged: app
[INFO] source: /home4/home/n02-puma/tetts/roses/u-db898/app
[INFO] unchanged: bin
[INFO] source: /home4/home/n02-puma/tetts/roses/u-db898/bin
[INFO] unchanged: meta
[INFO] source: /home4/home/n02-puma/tetts/roses/u-db898/meta
[INFO] unchanged: rose-suite.info
[INFO] source: /home4/home/n02-puma/tetts/roses/u-db898/rose-suite.info
[INFO] unchanged: site
[INFO] source: /home4/home/n02-puma/tetts/roses/u-db898/site
[INFO] unchanged: suite-tests-graph.rc
[INFO] source: /home4/home/n02-puma/tetts/roses/u-db898/suite-tests-graph.rc
[INFO] unchanged: suite-tests-runtime.rc
[INFO] source: /home4/home/n02-puma/tetts/roses/u-db898/suite-tests-runtime.rc
[INFO] REGISTERED u-db898 → /home/n02/n02/tetts/cylc-run/u-db898
[INFO] Valid for cylc-7.8.12
[FAIL] ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n ln03 env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-db898\ –run=run\ –remote=uuid=ab52ec80-a974-4fb0-9ab2-14e75f79fdcb,now-str=20250507T094601Z,root-dir='$DATADIR' # return-code=1, stderr=
[FAIL] This command has been replaced by: “cylc install”.
[tetts@puma2 ~]$ rose suite-run -v -v -C roses/u-db898
[INFO] 2025-05-07T10:50:26+0100 cylc get-global-config -i [hosts][localhost]run\ directory
[INFO] 2025-05-07T10:50:27+0100 cylc get-global-config -i [hosts][localhost]work\ directory
[INFO] 2025-05-07T10:50:27+0100 chdir: roses/u-db898/
[INFO] 2025-05-07T10:50:27+0100 Configuration: /home4/home/n02-puma/tetts/roses/u-db898/
[INFO] 2025-05-07T10:50:27+0100 file: rose-suite.conf
[INFO] 2025-05-07T10:50:27+0100 cylc --version
[INFO] 2025-05-07T10:50:27+0100 export CYLC_VERSION=7.8.12
[INFO] 2025-05-07T10:50:27+0100 export ROSE_ORIG_HOST=puma2.archer2.ac.uk
[INFO] 2025-05-07T10:50:27+0100 export ROSE_SITE=
[INFO] 2025-05-07T10:50:27+0100 export ROSE_VERSION=2019.01.3
[INFO] 2025-05-07T10:50:27+0100 create: log.20250507T095027Z
[INFO] 2025-05-07T10:50:27+0100 delete: log
[INFO] 2025-05-07T10:50:27+0100 symlink: log.20250507T095027Z <= log
[INFO] 2025-05-07T10:50:27+0100 tar -czf log.20250507T094601Z.tar.gz log.20250507T094601Z
[INFO] 2025-05-07T10:50:27+0100 log.20250507T094601Z.tar.gz <= log.20250507T094601Z
[INFO] 2025-05-07T10:50:27+0100 delete: log.20250507T094601Z/
[INFO] 2025-05-07T10:50:27+0100 create: log/suite
[INFO] 2025-05-07T10:50:27+0100 create: log/rose-conf
[INFO] 2025-05-07T10:50:27+0100 svn info --non-interactive
[INFO] 2025-05-07T10:50:27+0100 svn status --non-interactive
[INFO] 2025-05-07T10:50:27+0100 svn diff --internal-diff --non-interactive
[INFO] 2025-05-07T10:50:27+0100 git describe
[INFO] 2025-05-07T10:50:27+0100 symlink: rose-conf/20250507T105027-run.conf <= log/rose-suite-run.conf
[INFO] 2025-05-07T10:50:27+0100 symlink: rose-conf/20250507T105027-run.version <= log/rose-suite-run.version
[INFO] 2025-05-07T10:50:27+0100 fcm info --xml fcm:um.xm_tr/rose-stem/ana/mule_cumf.py@vn11.4
[INFO] 2025-05-07T10:50:28+0100 unchanged: ana/mule_cumf.py
[INFO] 2025-05-07T10:50:28+0100 source: svn://puma2.archer2.ac.uk/um.xm/main/trunk/rose-stem/ana/mule_cumf.py@71790 (fcm:um.xm_tr/rose-stem/ana/mule_cumf.py@vn11.4)
[INFO] 2025-05-07T10:50:28+0100 unchanged: app
[INFO] 2025-05-07T10:50:28+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/app
[INFO] 2025-05-07T10:50:28+0100 unchanged: bin
[INFO] 2025-05-07T10:50:28+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/bin
[INFO] 2025-05-07T10:50:28+0100 unchanged: meta
[INFO] 2025-05-07T10:50:28+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/meta
[INFO] 2025-05-07T10:50:28+0100 unchanged: rose-suite.info
[INFO] 2025-05-07T10:50:28+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/rose-suite.info
[INFO] 2025-05-07T10:50:28+0100 unchanged: site
[INFO] 2025-05-07T10:50:28+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/site
[INFO] 2025-05-07T10:50:28+0100 unchanged: suite-tests-graph.rc
[INFO] 2025-05-07T10:50:28+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/suite-tests-graph.rc
[INFO] 2025-05-07T10:50:28+0100 unchanged: suite-tests-runtime.rc
[INFO] 2025-05-07T10:50:28+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/suite-tests-runtime.rc
[INFO] 2025-05-07T10:50:28+0100 cylc register u-db898 /home/n02/n02/tetts/cylc-run/u-db898
[INFO] 2025-05-07T10:50:29+0100 REGISTERED u-db898 → /home/n02/n02/tetts/cylc-run/u-db898
[INFO] 2025-05-07T10:50:29+0100 cylc validate -o /tmp/tmp2chbgm --strict u-db898
[INFO] 2025-05-07T10:50:30+0100 Valid for cylc-7.8.12
[INFO] 2025-05-07T10:50:30+0100 cylc get-config -ao -i [remote]owner -i [remote]host u-db898
[INFO] 2025-05-07T10:50:31+0100 bash -ec H=$(rose\ host-select\ archer2);\ echo\ $H
[INFO] 2025-05-07T10:50:31+0100 ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n ln01 env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-db898\ –run=run\ –remote=uuid=01542424-c658-4f69-ae77-ef5f5adfcbd8,now-str=20250507T095027Z,root-dir='$DATADIR'
[FAIL] 2025-05-07T10:50:33+0100 ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n ln01 env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-db898\ –run=run\ –remote=uuid=01542424-c658-4f69-ae77-ef5f5adfcbd8,now-str=20250507T095027Z,root-dir='$DATADIR' # return-code=1, stderr=
[FAIL] 2025-05-07T10:50:33+0100 This command has been replaced by: “cylc install”.
[tetts@puma2 ~]$

Hi Simon,

What cylc version are you using? did you load cylc 8 before running this or another suite? (export CYLC_VERSION=8)

Can you run cylc --version on Puma2 to check that you’re using Cylc 7 as expected? Your suite is set up for Cylc 7 and if Puma2 is picking up Cylc 8 by default, it could explain the issue you’re seeing when you run rose suite-run.

Juan

Hi Juan,
I am running version 7.8.12
[tetts@puma2 ~]$ cylc --version
7.8.12
Simon

Thanks Simon, can you please run cylc --version and inmidiatly the command that run before rose suite-run -v -C roses/u-db898 and share with us that you get.

I’ve given you the debug info. It says inside it that I am running cylc version 7. And all I did before was logon. So I am confused what you want me to do. I suspect it is something to do with my environment. Anyhow here you are:
(conda)stett2@scotia ~ $ ssh tetts@login.archer2.ac.uk
Last login: Wed May 7 10:03:26 2025 from 129.215.8.165
#######################################################################################

    @@@@@@@@@
 @@@         @@@            _      ____     ____   _   _   _____   ____    ____

@@@ @@@@@ @@@ / \ | _ \ / | | | | | | | | _ \ |
@@@ @@ @@ @@@ / _ \ | |
) | | | | |
| | | | | |) | ) |
@@ @@ @@@ @@ @@ / ___ \ | _ < | |
| _ | | |
__ | _ < / /
@@ @@ @@@ @@ @@ // _\ || _\ _
| || || |_____| || _\ |_____|
@@@ @@ @@ @@@
@@@ @@@@@ @@@ Support and Access
@@@ @@@
@@@@@@@@@

  •     U K R I         -        E P C C        -         H P E   C r a y         -
    

Hostname: ln01
Distribution: SLES 15.4 4
CPUS: 256
Memory: 515.3GB
Configured: 2025-04-16

######################################################################################
---------------------------------Welcome to ARCHER2-----------------------------------
######################################################################################

tetts@ln01:~> ssh -Y puma2
######################################################################################
---------------------------------Welcome to PUMA2-----------------------------------
######################################################################################
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Tue May 6 20:07:30 2025 from 172.24.75.51
Met Office Science Repository Service password:
svn: E170013: Unable to connect to a repository at URL ‘https://code.metoffice.gov.uk/svn/test
svn: E215004: No more credentials or we tried too many times.
Authentication failed
Error: Unable to access Subversion with given password
Run “mosrs-cache-password” to try caching your password again
[tetts@puma2 ~]$ logout
Connection to puma2 closed.
tetts@ln01:~> ssh -Y puma2
######################################################################################
---------------------------------Welcome to PUMA2-----------------------------------
######################################################################################
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Wed May 7 16:51:36 2025 from 172.24.75.51
Met Office Science Repository Service password:
No password entered - abort
^A^ERun “mosrs-cache-password” to try caching your password again
[tetts@puma2 ~]$
[tetts@puma2 ~]$
[tetts@puma2 ~]$ logout
Connection to puma2 closed.
tetts@ln01:~> ssh -Y puma2
######################################################################################
---------------------------------Welcome to PUMA2-----------------------------------
######################################################################################
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Wed May 7 16:51:49 2025 from 172.24.75.51
[tetts@puma2 ~]$ cylc --version
7.8.12
[tetts@puma2 ~]$ rose suite-run -v -v -C roses/u-db898
[INFO] 2025-05-07T16:53:34+0100 cylc get-global-config -i [hosts][localhost]run\ directory
[INFO] 2025-05-07T16:53:35+0100 cylc get-global-config -i [hosts][localhost]work\ directory
[INFO] 2025-05-07T16:53:35+0100 chdir: roses/u-db898/
[INFO] 2025-05-07T16:53:35+0100 Configuration: /home4/home/n02-puma/tetts/roses/u-db898/
[INFO] 2025-05-07T16:53:35+0100 file: rose-suite.conf
[INFO] 2025-05-07T16:53:35+0100 cylc --version
[INFO] 2025-05-07T16:53:35+0100 export CYLC_VERSION=7.8.12
[INFO] 2025-05-07T16:53:35+0100 export ROSE_ORIG_HOST=puma2.archer2.ac.uk
[INFO] 2025-05-07T16:53:35+0100 export ROSE_SITE=
[INFO] 2025-05-07T16:53:35+0100 export ROSE_VERSION=2019.01.3
[INFO] 2025-05-07T16:53:35+0100 create: log.20250507T155335Z
[INFO] 2025-05-07T16:53:35+0100 delete: log
[INFO] 2025-05-07T16:53:35+0100 symlink: log.20250507T155335Z <= log
[INFO] 2025-05-07T16:53:35+0100 tar -czf log.20250507T095027Z.tar.gz log.20250507T095027Z
[INFO] 2025-05-07T16:53:35+0100 log.20250507T095027Z.tar.gz <= log.20250507T095027Z
[INFO] 2025-05-07T16:53:35+0100 delete: log.20250507T095027Z/
[INFO] 2025-05-07T16:53:35+0100 create: log/suite
[INFO] 2025-05-07T16:53:35+0100 create: log/rose-conf
[INFO] 2025-05-07T16:53:35+0100 svn info --non-interactive
[INFO] 2025-05-07T16:53:35+0100 svn status --non-interactive
[INFO] 2025-05-07T16:53:35+0100 svn diff --internal-diff --non-interactive
[INFO] 2025-05-07T16:53:36+0100 git describe
[INFO] 2025-05-07T16:53:36+0100 symlink: rose-conf/20250507T165335-run.conf <= log/rose-suite-run.conf
[INFO] 2025-05-07T16:53:36+0100 symlink: rose-conf/20250507T165335-run.version <= log/rose-suite-run.version
[INFO] 2025-05-07T16:53:36+0100 fcm info --xml fcm:um.xm_tr/rose-stem/ana/mule_cumf.py@vn11.4
[INFO] 2025-05-07T16:53:37+0100 unchanged: ana/mule_cumf.py
[INFO] 2025-05-07T16:53:37+0100 source: svn://puma2.archer2.ac.uk/um.xm/main/trunk/rose-stem/ana/mule_cumf.py@71790 (fcm:um.xm_tr/rose-stem/ana/mule_cumf.py@vn11.4)
[INFO] 2025-05-07T16:53:37+0100 unchanged: app
[INFO] 2025-05-07T16:53:37+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/app
[INFO] 2025-05-07T16:53:37+0100 unchanged: bin
[INFO] 2025-05-07T16:53:37+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/bin
[INFO] 2025-05-07T16:53:37+0100 unchanged: meta
[INFO] 2025-05-07T16:53:37+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/meta
[INFO] 2025-05-07T16:53:37+0100 unchanged: rose-suite.info
[INFO] 2025-05-07T16:53:37+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/rose-suite.info
[INFO] 2025-05-07T16:53:37+0100 unchanged: site
[INFO] 2025-05-07T16:53:37+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/site
[INFO] 2025-05-07T16:53:37+0100 unchanged: suite-tests-graph.rc
[INFO] 2025-05-07T16:53:37+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/suite-tests-graph.rc
[INFO] 2025-05-07T16:53:37+0100 unchanged: suite-tests-runtime.rc
[INFO] 2025-05-07T16:53:37+0100 source: /home4/home/n02-puma/tetts/roses/u-db898/suite-tests-runtime.rc
[INFO] 2025-05-07T16:53:37+0100 cylc register u-db898 /home/n02/n02/tetts/cylc-run/u-db898
[INFO] 2025-05-07T16:53:37+0100 REGISTERED u-db898 → /home/n02/n02/tetts/cylc-run/u-db898
[INFO] 2025-05-07T16:53:37+0100 cylc validate -o /tmp/tmp25qO2t --strict u-db898
[INFO] 2025-05-07T16:53:38+0100 Valid for cylc-7.8.12
[INFO] 2025-05-07T16:53:38+0100 cylc get-config -ao -i [remote]owner -i [remote]host u-db898
[INFO] 2025-05-07T16:53:39+0100 bash -ec H=$(rose\ host-select\ archer2);\ echo\ $H
[INFO] 2025-05-07T16:53:40+0100 ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n ln03 env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-db898\ –run=run\ –remote=uuid=22877562-a68a-4f9a-943a-2878231d0b21,now-str=20250507T155335Z,root-dir='$DATADIR'
[FAIL] 2025-05-07T16:53:42+0100 ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8 -n ln03 env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-db898\ –run=run\ –remote=uuid=22877562-a68a-4f9a-943a-2878231d0b21,now-str=20250507T155335Z,root-dir='$DATADIR' # return-code=1, stderr=
[FAIL] 2025-05-07T16:53:42+0100 This command has been replaced by: “cylc install”.