Replacing id_rsa_archerum

Hello,
I think I have never actually run a UM suite from puma to archer2 and did not have an id_rsa_archerum in pumanew ~/.ssh.
I have just created a fresh one and the help pages say it will be auto detected in the next couple of days. I have also done this:
markr@pumanew:~/.ssh>ssh-add ./id_rsa_archerum
Enter passphrase for ./id_rsa_archerum:
Identity added: ./id_rsa_archerum (./id_rsa_archerum)
markr@pumanew:~/.ssh>

Do I have to do anything else? (markr on pumanew and mricha on archer2) I did this because the suite failed to launch. /home/markr/roses/u-ct529 perhaps I should try a simpler suite first.

thanks,
MarkR

Hi Mark,

Your id_rsa_archerum key has now been added to ARCHER2. Please check you get the following response when logging to ARCHER2 from puma.

ros@pumanew$ ssh login.archer2.ac.uk
PTY allocation request failed on channel 0
Comand rejected by policy. Not in authorised list 
Connection to login.archer2.ac.uk closed.

Regards,
Ros.

Hi Ros,

Yes I go the message you indicated but rose suite-run still fails.

[FAIL] ssh -oBatchMode=yes -n mricha@login.archer2.ac.uk env\ ROSE_VERSION=2019.01.3\ CYLC_VERSION=7.8.12\ bash\ -l\ -c\ '"$0"\ "$@"'\ rose\ suite-run\ -vv\ -n\ u-ct529\ –run=run\ –remote=uuid=bc3521ee-54fc-4d6e-a889-bb8d3ae2a657,now-str=20230118T120306Z,root-dir='$DATADIR' # return-code=255, stderr=

[FAIL] Host key verification failed.

markr@pumanew:~/roses/u-ct529>ssh-add -l

4096 SHA256:mniZil9oDVQTvD/Is10l49iREMINH2WXIPK5c8O2NrU ./id_rsa_archerum (RSA)

image001.jpg

Hi Mark,

Please try logging into each login nodes 1-4 (e.g. login1.archer2.ac.uk). At least one of them will list a dodgey key that will need removing from your ~/.ssh/known_hosts file.

Regards,
Ros.

Hi Ros,
I just the ssh command tried again to make it match the rose suite command:
markr@pumanew:~/roses/u-ct529>ssh mricha@login.archer2.ac.uk
Warning: the ECDSA host key for ‘login.archer2.ac.uk’ differs from the key for the IP address ‘193.62.216.42’
Offending key for IP in /home/markr/.ssh/known_hosts:6
Matching host key in /home/markr/.ssh/known_hosts:18
Are you sure you want to continue connecting (yes/no)? yes
PTY allocation request failed on channel 0
Comand rejected by policy. Not in authorised list
Connection to login.archer2.ac.uk closed.

then the suite again still fails like this:
[FAIL] Host key verification failed.

Remove the offending key and try again - there may be more than one offending key.

Offending key for IP in /home/markr/.ssh/known_hosts:6

Well there are several similar IP in the known hosts 192.62.216.4* so maybe I have to delete all those as “login.archer2.ac.uk” is likely to be several machines

markr@pumanew:~/roses/u-ct529>ssh mricha@login.archer2.ac.uk
The authenticity of host ‘login.archer2.ac.uk (193.62.216.45)’ can’t be established.
ECDSA key fingerprint is SHA256:UGS+LA8I46LqnD58WiWNlaUFY3uD1WFr+V8RCG09fUg.
ECDSA key fingerprint is MD5:71:2a:aa:7a:b1:a5:43:03:d2:ec:4a:da:5f:c0:0f:8c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘login.archer2.ac.uk’ (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for ‘login.archer2.ac.uk’ differs from the key for the IP address ‘193.62.216.45’
Offending key for IP in /home/markr/.ssh/known_hosts:2
Are you sure you want to continue connecting (yes/no)? yes
PTY allocation request failed on channel 0
Comand rejected by policy. Not in authorised list
Connection to login.archer2.ac.uk closed.

Yes, indeed. Hence my comment above about logging into each ARCHER2 login node.

oops missed that message! will delete all those 62.4* and try login1-4 and then tray suite.

Hmm,
no progress yet. the archer2 IPS reoved and logins1 to login4 ssh’d as expected (I can even do an ls command)
markr@pumanew:~/.ssh>ssh mricha@login4.archer2.ac.uk ls
bin
BUILD
cesm_prep
ck-bin
cylc-run
Docker
Experimental
later_dot_cime
monc_module_prep_202207.txt
mr_dot_cime_fragments
mrigrep.txt
my_modules
NOTES
Projects
PythonHPC.zip
SingWork
SW
TAR_files

but no joy from suite u-ct529

not sure where to look next. the id_rsa_archerum was generated fresh last night and I used the ssh-setup so that th ssh-add shows it is live. also the “ls” succeeded. I could send you a "ssh -v -v -v " log if it helps.

markr

Does ssh mricha@login.archer2.ac.uk give the expected response too?

Are you retriggering the task or doing a rose suite-run? If the former, try stopping the suite and starting it again. Sometimes cylc doesn’t pickup the new ssh changes.

Yeah I’m in:

markr@pumanew:~/.ssh>grep archer known_hosts
login1.archer2.ac.uk,193.62.216.42 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnMeFf1TPZ4pbupWeD4IeahEeeqJMAhrCv1znyQGAL45yOIArVltscW8GNhzfaWk5vKb9sIAm2mJZPc3b7te3c=
login2.archer2.ac.uk,193.62.216.43 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnMeFf1TPZ4pbupWeD4IeahEeeqJMAhrCv1znyQGAL45yOIArVltscW8GNhzfaWk5vKb9sIAm2mJZPc3b7te3c=
login3.archer2.ac.uk,193.62.216.44 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnMeFf1TPZ4pbupWeD4IeahEeeqJMAhrCv1znyQGAL45yOIArVltscW8GNhzfaWk5vKb9sIAm2mJZPc3b7te3c=
login4.archer2.ac.uk,193.62.216.45 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnMeFf1TPZ4pbupWeD4IeahEeeqJMAhrCv1znyQGAL45yOIArVltscW8GNhzfaWk5vKb9sIAm2mJZPc3b7te3c=
markr@pumanew:~/.ssh>ssh mricha@login.archer2.ac.uk ls
The authenticity of host ‘login.archer2.ac.uk (193.62.216.42)’ can’t be established.
ECDSA key fingerprint is SHA256:UGS+LA8I46LqnD58WiWNlaUFY3uD1WFr+V8RCG09fUg.
ECDSA key fingerprint is MD5:71:2a:aa:7a:b1:a5:43:03:d2:ec:4a:da:5f:c0:0f:8c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘login.archer2.ac.uk’ (ECDSA) to the list of known hosts.
bin
BUILD
cesm_prep
ck-bin
cylc-run
Docker
Experimental
later_dot_cime
monc_module_prep_202207.txt
mr_dot_cime_fragments
mrigrep.txt
my_modules
NOTES
Projects
PythonHPC.zip
SingWork
SW
TAR_files

A weakness of “known_hosts” I think. DNS lookup and multiple identity for an alias “login.archer2.ac.uk” added to complications. Suite has started and Gcyls showing correctly.

thanks for the help and good suggestions
Mark