OpenSSH Certificate Authentication#

The problem with host keys#

It is very convenient to allocate part of a node ore even a full node for interactive work on Levante. We describe how to do this in our main documentation.

If you always use a login node of Levante to log into the allocated node, then you won’t have any problems with host keys. However, if you use a login node as a jump host for loging into the allocated node from your local machine, then you may see a warning.

$ ssh -J
The authenticity of host ' (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:BP1WmG35fIvIN+UPMorv5qxXtjaSkx7v08Dtc5kAteM.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Or even this dreaded error.

$ ssh -J
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

Internal nodes of Levante have individual host keys which are different from the login nodes. They also change frequently which has to do with the way our vendor deploys the software on these nodes.

You might ask why use a jump host at all if it only creates problems? Well, for some applications it is the only sensible way to go. VS Code remote development for instance wants to connect directly to the host where you do remote work. Since allocated nodes cannot be accessed from the outside, you have to use a jump host. The start-vnc script we provide for remote desktop also uses a jump host.

So what can be done? One option would be to remove the offending key from your ~/.ssh/known_hosts file and confirm the fingerprint of the host key on Levante before accepting it.

$ ssh-keyscan l40051 | ssh-keygen -lf -

This will work until the host key of the node is rotated again, at which point you have to start over.

Configuring your openssh client#

A more permanent solution is to configure your OpenSSH client in a similar way as Levante’s nodes are configured, because there you don’t have to worry about rotated host keys at all.

Trust the certificate authority (CA)#

To do that, add the public key of Levante’s CA to your ~/.ssh/known_hosts

echo "@cert-authority  *  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCf8OhyGgbXVVszhLLXxLpq84/2x3mmOinrPyuBau/sMkxvgdJ4Td8ApDdywYdwEiRyq4QDgoDSLwHMsT8tlRiRSms3cZCCqNsuRTGwyi54poRr9dQQ1WVwMFoBZoRWFxhQ0Vl1UJaG1HQ0URMQfRu+IyVy5Q6h1ico+BuB41hQz3xKyGfwWZMbB4vBp8NCnLK9yyEICKPKHxLmVFNHBqA41pTRA+9lEiYUnpeq405D9tPrv9H8DFd3J2B9QjjTTbgytfIJzaabvQFshtIVhTYyu8RHeBAIhl2FN/LgSBFyLABH5Ul5MNmxzvcnfw2GdZIqwTQ0dxDToM5oX6d1YB7whOM2WECfVC145hsjM/ct1ltTrftDfaJRJqp+U7RIVBfNVJDFhuF5z2k8kzqpjXS8/Ja8wsK+41yPPp6No1/HkM6a1b+ym0Xpb8+QdUr3VpFUQ6mMXMVVw4IM09iilckAcXWiu8H9CmCId2wQQ5CecTdFDnJaf1YNJBTbi24eCAIbQig/gocIhkBO7hBbf1UJ03Nvf7oqPi02afk5cSKfGdYPMQ0jsG3SvaJIFzJxMxUHHvT6+BqKqn0cGpwgwpHRvU49pxdrJLyZ4S/WOOhAYgftioDtaPv5haRTeQe36Y+/Z9ISf48YBKCxmwOR/7Ct0e9vY2pvIRavxxKn9DFioQ==" >> ~/.ssh/known_hosts

This means that your openssh client now trusts all host keys in * which were signed by our CA.

Config change#

Unfortunately, this was not all you need. At least not if you use a recent openssh on a recent Linux distribution.

You should check if you actually need the change. First delete all host keys for hosts matching * from your ~/.ssh/known_hosts. If you are able to log into an allocated node without a warning that the host key is unknown,

$ ssh -J
Last login: Tue Apr 11 12:19:49 2023 from
[l40051 ~] $

then no further configuration is needed. If openssh still asks to accept the key, then make sure your ~/.ssh/config contains the line CASignatureAlgorithms +ssh-rsa. Just create ~/.ssh/config as shown below if it doesn’t exist.

Host *
    CASignatureAlgorithms +ssh-rsa

Now you should be able to log in without any further prompts to accept a host key. Even when the host keys are rotated, you should not be bothered. Actually, your client now doesn’t store those host keys. It just checks that the key was signed and leaves your ~/.ssh/known_hosts uncluttered. At least not with keys of Levante nodes.