OpenSSH Certificate Authentication

The problem with host keys

You have probably seen the message openssh throws at you when you try to log into a new and unknown host.

$ ssh levante.dkrz.de
The authenticity of host 'levante (136.172.124.5)' can't be established.
RSA key fingerprint is SHA256:r95ZOIATUgcJJYCVRPDp4irsQzr1z7IlRN8UgU8i1vk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Or even this dreaded error that seemingly comes out of nowhere.

$ ssh levante.dkrz.de
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It all boils down to one problem: how do you know that you are actually connecting to Levante and not to some other system just posing as Levante. The traditional way was to compare the fingerprint of the host’s key to a fingerprint which you got by a trusted source. But really, who did that anyway?

Then we sometimes have to change the host keys. Actually, we should do that on a regular basis because host keys can be stolen just like user keys. But when you change the host key, users who already logged into Levante in the past get the alarming sounding second message.

So what can be done? OpenSSH came up with a way to sign ssh keys. Now you only need to trust the entity that signed the keys for Levante and not each individual host key.

Configuring your openssh client

Of course, this doesn’t work right out of the box. The upside of the configuration is that you only have to do it once and you won’t be bothered by unknown or changed host keys for Levante in the future. In fact, if you still get one of the above messages with Levante nodes, you should find the reason for them before connecting.

Trust the certificate authority (CA)

To do that, add the public key of Levante’s CA to your ~/.ssh/known_hosts

echo "@cert-authority *.dkrz.de ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8RIUxOll18o4dqZlpSaJGw+GJ4yx+6L04L3uxdgHf+cKELUx9tiZoVTJ7hpC/YE8Mlra9rKXILZ7PvjeMiytG/7wnRzC/MPY0k95MEAQP9gWXGidNgqXq7lAZ6xNAJeEXKYhi/5VdhqCMwgM6qgfDs4N81Dpq3N7sXi4kNuGY8yRwoGGPK9eyq5vgmupMDcJyV8pUpBan1EfZgr4bly7LLL/GqrE0E/bPjjGxNJs7wM7Do68qxn89POqlc4DxbA3yogULitvhzRo0Xoif1YpVcMTN/ptw6ERJ9C2jIr/HTB+xUt2AtdeST7xX0yITsKVgrMuDLTOPT3rr6BIRSJGRSH5I1FfuaPCRWouFV5VQVd0Turd45qi9uUretxZy7xTgETkYES/Cpu0O3Q9qhIzhwKoSyqudSV4uqeWCWcSwASZ8laXYT959zsZ684Dl+r6JslpV9/W88JMWeFmPl3/r9A62DkDPPV+vF118VpyKygjAfzEVwvfKjUoXLu4MObDClVwWTQdorIzxd4aNcUr768lHoXoggmyASY4kKnDkBL+mwSrjWAavJ56FYJOxVtA1+I/qcSp3r+nRgYHWHtt8N80ZNST3Btfbnr5n87oy5SDGJBHcb45rvBZR3sVdaSqHeov8BQp3yyI+NejSAHBXKu/oapskWDdxhhqWKKbI0Q==" >> ~/.ssh/known_hosts

This means that your openssh client now trusts all host keys in *.dkrz.de which were signed by our CA.

Config change

Unfortunately, this was not all you need. At least not if you use a recent openssh on a recent Linux distribution. While Levante is a new system, it already uses an outdated signature algorithm. We hope that Levante’s vendor will be able to change that.

You should check if you actually need the change. First delete all Levante host keys from your ~/.ssh/known_hosts. If you are able to log in without a warning that the host key is unknown,

$ ssh levante.dkrz.de
Last login: Sun Feb 21 13:56:48 2022 from xxx.xxx.xxx.xxx

then no further configuration is needed. If openssh still asks to accept the key then make sure your ~/.ssh/config contains the line CASignatureAlgorithms +ssh-rsa. Just create ~/.ssh/config as shown below if it doesn’t exist.

Host *.dkrz.de
    CASignatureAlgorithms +ssh-rsa

At this point you should be able to log in without any further prompts to accept a host key. Even if we change the host key, you should not be bothered. Actually, your client now doesn’t store the host keys. It just checks that the key was signed and leaves your ~/.ssh/known_hosts uncluttered. At least not with keys of Levante nodes.