OpenSSH Certificate Authentication#
The problem with host keys#
It is very convenient to allocate part of a node ore even a full node for interactive work on Levante. We describe how to do this in our main documentation.
If you always use a login node of Levante to log into the allocated node, then you won’t have any problems with host keys. However, if you use a login node as a jump host for loging into the allocated node from your local machine, then you may see a warning.
$ ssh -J levante.dkrz.de l40051.lvt.dkrz.de
The authenticity of host 'l40051.lvt.dkrz.de (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:BP1WmG35fIvIN+UPMorv5qxXtjaSkx7v08Dtc5kAteM.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Or even this dreaded error.
$ ssh -J levante.dkrz.de l40051.lvt.dkrz.de
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
Internal nodes of Levante have individual host keys which are different from the login nodes. They also change frequently which has to do with the way our vendor deploys the software on these nodes.
You might ask why use a jump host at all if it only creates problems?
Well, for some applications it is the only sensible way to go. VS Code
remote development for instance wants to connect directly to the host
where you do remote work. Since allocated nodes cannot be accessed
from the outside, you have to use a jump host. The start-vnc script
we provide for remote desktop also uses a jump host.
So what can be done? One option would be to remove the offending key
from your ~/.ssh/known_hosts file and confirm the fingerprint of
the host key on Levante before accepting it.
$ ssh-keyscan l40051 | ssh-keygen -lf -
This will work until the host key of the node is rotated again, at which point you have to start over.
Configuring your openssh client#
A more permanent solution is to configure your OpenSSH client in a similar way as Levante’s nodes are configured, because there you don’t have to worry about rotated host keys at all.
Config change#
Unfortunately, this was not all you need. At least not if you use a recent openssh on a recent Linux distribution.
You should check if you actually need the change. First delete all
host keys for hosts matching *.lvt.dkrz.de from your
~/.ssh/known_hosts. If you are able to log into an allocated node
without a warning that the host key is unknown,
$ ssh -J levante.dkrz.de l40051.lvt.dkrz.de
Last login: Tue Apr 11 12:19:49 2023 from 10.9.0.1
[l40051 ~] $
then no further configuration is needed. If openssh still asks to
accept the key, then make sure your ~/.ssh/config contains the line
CASignatureAlgorithms +ssh-rsa. Just create ~/.ssh/config
as shown below if it doesn’t exist.
Host *.lvt.dkrz.de
CASignatureAlgorithms +ssh-rsa
Now you should be able to log in without any further prompts to accept
a host key. Even when the host keys are rotated, you should not be
bothered. Actually, your client now doesn’t store those host keys. It
just checks that the key was signed and leaves your
~/.ssh/known_hosts uncluttered. At least not with keys of Levante
nodes.