> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution.
Maybe I'm missing something but SSH already has a built-in solution for this, key-certs. Just sign the server key with a private CA key you trust.
washingupliquid 1 hours ago [-]
I'm supposed to believe MitM with the same exact keypair is somehow possible? Private keys are never exchanged. Did everybody forget how crypto works?
Yes you implicitly trust the public key on first login.... then just... immediately compare it with what's on your box?
Might as well seal your doors with duct tape to prevent ghosts from entering your home because this is equally effective.
leni536 44 minutes ago [-]
How do you compare? What trusted channel do you use to retrieve the real public key?
londons_explore 5 hours ago [-]
A big class of attacker is nation state attackers who do not want to risk discovery.
A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.
For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.
Suddenly, nation state attackers won't be interested in MITM-ing at all.
Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
hnlmorg 5 hours ago [-]
Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
If so, the legitimate server wouldn’t have anything in their logs that would help detect such an attack.
OpenSSH does log other telemetry though.
cesarb 57 minutes ago [-]
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
The client sends not only the public key, but also a signature, and that signature depends on the output from the key exchange, so it's "bound" to the shared keys negotiated between the client and the server. If the MITM server does separate key exchanges with the client (pretending to be the real server) and the server (pretending to be the real client), the signature won't match; if it forwards the key exchange between the real client and the real server, it won't be able to decrypt the packets.
That's the best thing about SSH public key authentication (and HTTPS client certificates): even when MITM can impersonate the server to the client (because the client didn't verify the host key), it can't impersonate the client to the real server.
mbxy 15 minutes ago [-]
Let me see if I understand correctly: Client takes its own public key and the server's public key and creates this signature.
MITM can take its public key and the client's public key and send the resulting signature to the server instead of forwarding what it received from the client.
Do pretty much the same exact thing: MITM PK + Server's PK -> Client. Now client has a signature as well. The signatures that client and server have are different but that is OK as long as MITM can see and change all communication.
It has been a while since I went through the details of the protocol, so I must be missing something. What is it?
flumpcakes 4 hours ago [-]
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
Fingerprints are derived from the certificates/private keys. Unless I don't understand some basic crypto, or SSH works in some obtuse way, I do not think it would be possible for the MITM attacker to present the server with the true client's fingerprint unless they also had had the client's private key.
hnlmorg 3 hours ago [-]
Ah thanks for the explanation. It’s been a long long time since I’ve delved this deeply into the topic.
daveguy 2 hours ago [-]
Because log processing is handled in the kernel/root/system? Is this a trick question?
See also: rsyslogd
sieabahlpark 3 hours ago [-]
[dead]
INTPenis 4 hours ago [-]
The author essentially bootstraps their servers with a known trusted host key, so that first connection is recognized, instead of having to trust a new and recently generated host key when you first connect.
It's a neat little trick if you're often deploying VPS in shared cloud environments.
flumpcakes 4 hours ago [-]
This is something I have struggled with:
How to deploy secrets during bootstrap to a new virtual machine running in the Cloud that does not leave a trace in the infrastructure. And in a way that I can completely automate the deployment.
One answer is providing the secrets in cloudinit - but this leaves a trail on the host/provider's infrastructure, I do not know if those configs I paste into the portal then get saved off somewhere.
The other option (more secure) is having the keys/secrets generated on the host itself at first boot. But then this is difficult to automate as I would need to scrap them (even just the public parts) in a secure way. One option would be to have the public keys printed to the terminal/VNC - but this is much more trouble than it is worth to automate.
I'm not sure on a good solution. This is taking quite an adversarial security model though, assuming the host/provider is not completely trustworthy. Of course not owning the hardware means that the host/provider could be performing other attacks without my knowledge (copying memory, etc.)
unsnap_biceps 3 hours ago [-]
I've been meaning to investigate Nitro Enclaves[1] for exactly this but haven't made the time yet. Have you looked into them and found them lacking?
1. Use cloudinit but give it a one time password to download the secrets on first boot.
2. Use certificates and your own CA.
3. Use the virtual serial console for first login.
4. Use cloudinit to add a custom software repo, then use that to install a custom package that does the initial work.
anygivnthursday 1 hours ago [-]
You can bootstrap from your custom ISO with some embedded starter key, upload ISO, loads into RAM and opens SSH, connect and run a playbook to encrypt the root drive where you deploy your OS with your SSH key. If you went with encrypted root, you might need to enter pass via console, or dropbear server in a pre-boot env you connect to via SSH to enter the key if you dont trust the console, or setup some custom network unlock mechanism, etc. But once unlocked your provider can still dump your keys from memory. There are also things like AMD SEV-SNP for some more confidential use cases.
asalahli 3 hours ago [-]
It baffles me why VPS providers don't display the fingerprints on their dashboards or expose them through the API.
dspillett 47 minutes ago [-]
A few do. Though a more common way of securing the loop is to take the desired user public key during the build process.
Assuming you trust that the host control panel (or API server) hasn't been hacked, which you are assuming anyway if you trust a host fingerprint given to you that way, that should be as secure. For a small bit of extra assurance, to protect from an extra very unlikely attack, generate a new key pair just for this VM's creation so that you know you aren't connecting to some other VM that happens to have a known public key of yours and you've been redirected to by DNS poisoning.
athrowaway3z 2 hours ago [-]
So this is a tangent on a thought i had after reading the title, but it might be a cool idea that I'll not have the time to do anything with so feel free to use it:
Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.
But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.
You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.
lvlabguy 3 hours ago [-]
MitM is not possible if one uses public key authentication.
gruez 3 hours ago [-]
I was about to downvote this for being obviously false, but after some research this does appear to be true, because ssh uses some channel binding mechanism to prevent your public key authentication from being replayed/reused by the "man" in the middle.
ekr____ 2 hours ago [-]
This is one of those situations where it's necessary to be very precise about the security properties.
Specifically, if you bind authentication to the connection, then an attacker who impersonates the server (in this case because it's the first connection, but in other settings because they have a fake certificate), then client authentication is not portable to another connection, so the attacker can't mount a classic MITM attack. However -- and this is a big however -- that doesn't mean that there aren't serious security problems. For example:
* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.
* If you download some file (e.g., a script) from the server and then trust it, the attacker can use that to provide a malicious script.
gruez 2 hours ago [-]
>* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.
That's much harder to pull off though, because you need to replicate the environment close enough so that the victim doesn't suspect anything. Do they put their config files in /var/lib or random docker volumes? Do they use docker compose or docker-compose, etc.
ekr____ 2 hours ago [-]
Sure. I'm not saying it's not better to use public key authentication (it is!). Just that it's still possible to have problems.
crypt0r84 4 hours ago [-]
provision the hosts with an SSH CA, use the CA as a trust root in openssh.
they are various version out there from the big players.
IgorPartola 3 hours ago [-]
There was ages ago a project called monkeysphere that let you sign the host ssh key with your gpg key and verify it automatically. The downside was that it was very slow.
zuzululu 3 hours ago [-]
but how real is this threat? you buy a box from hetzner or some VPS provider and its rooted when you try to ssh ?
projektfu 2 hours ago [-]
You get a box and someone is sitting on its IP address (in the middle), proxying to the real one, so everything is getting logged. Other comments say that this mitm stops working when you use public key authentication.
cesarb 54 minutes ago [-]
> Other comments say that this mitm stops working when you use public key authentication.
It doesn't completely stop working; a MITM can still pretend to be the server, it just can't authenticate to the real server on your behalf. You could be doing all your work in a fake server controlled by the attacker, while the real server sits there untouched.
projektfu 41 minutes ago [-]
If that were the case then I don't think the OP approach works either.
riffic 2 hours ago [-]
zero touch prod might be something folks should keep as a north star - SSH should be break glass and a last resort if possible.
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).
To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.
It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).
Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.
PunchyHamster 3 hours ago [-]
Seems like way easier way would be sshing for the first time and just typing `sudo reboot`. If VM reboots, it is yours
Or cat-ing some secrets that would be on target machine but not attacker
Rendered at 23:16:00 GMT+0000 (Coordinated Universal Time) with Vercel.
Maybe I'm missing something but SSH already has a built-in solution for this, key-certs. Just sign the server key with a private CA key you trust.
Yes you implicitly trust the public key on first login.... then just... immediately compare it with what's on your box?
Might as well seal your doors with duct tape to prevent ghosts from entering your home because this is equally effective.
A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.
For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.
Suddenly, nation state attackers won't be interested in MITM-ing at all.
Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
If so, the legitimate server wouldn’t have anything in their logs that would help detect such an attack.
OpenSSH does log other telemetry though.
The client sends not only the public key, but also a signature, and that signature depends on the output from the key exchange, so it's "bound" to the shared keys negotiated between the client and the server. If the MITM server does separate key exchanges with the client (pretending to be the real server) and the server (pretending to be the real client), the signature won't match; if it forwards the key exchange between the real client and the real server, it won't be able to decrypt the packets.
That's the best thing about SSH public key authentication (and HTTPS client certificates): even when MITM can impersonate the server to the client (because the client didn't verify the host key), it can't impersonate the client to the real server.
MITM can take its public key and the client's public key and send the resulting signature to the server instead of forwarding what it received from the client.
Do pretty much the same exact thing: MITM PK + Server's PK -> Client. Now client has a signature as well. The signatures that client and server have are different but that is OK as long as MITM can see and change all communication.
It has been a while since I went through the details of the protocol, so I must be missing something. What is it?
Fingerprints are derived from the certificates/private keys. Unless I don't understand some basic crypto, or SSH works in some obtuse way, I do not think it would be possible for the MITM attacker to present the server with the true client's fingerprint unless they also had had the client's private key.
See also: rsyslogd
It's a neat little trick if you're often deploying VPS in shared cloud environments.
How to deploy secrets during bootstrap to a new virtual machine running in the Cloud that does not leave a trace in the infrastructure. And in a way that I can completely automate the deployment.
One answer is providing the secrets in cloudinit - but this leaves a trail on the host/provider's infrastructure, I do not know if those configs I paste into the portal then get saved off somewhere.
The other option (more secure) is having the keys/secrets generated on the host itself at first boot. But then this is difficult to automate as I would need to scrap them (even just the public parts) in a secure way. One option would be to have the public keys printed to the terminal/VNC - but this is much more trouble than it is worth to automate.
I'm not sure on a good solution. This is taking quite an adversarial security model though, assuming the host/provider is not completely trustworthy. Of course not owning the hardware means that the host/provider could be performing other attacks without my knowledge (copying memory, etc.)
[1] https://aws.amazon.com/ec2/nitro/nitro-enclaves/
2. Use certificates and your own CA.
3. Use the virtual serial console for first login.
4. Use cloudinit to add a custom software repo, then use that to install a custom package that does the initial work.
Assuming you trust that the host control panel (or API server) hasn't been hacked, which you are assuming anyway if you trust a host fingerprint given to you that way, that should be as secure. For a small bit of extra assurance, to protect from an extra very unlikely attack, generate a new key pair just for this VM's creation so that you know you aren't connecting to some other VM that happens to have a known public key of yours and you've been redirected to by DNS poisoning.
Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.
But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.
You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.
Specifically, if you bind authentication to the connection, then an attacker who impersonates the server (in this case because it's the first connection, but in other settings because they have a fake certificate), then client authentication is not portable to another connection, so the attacker can't mount a classic MITM attack. However -- and this is a big however -- that doesn't mean that there aren't serious security problems. For example:
* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.
* If you download some file (e.g., a script) from the server and then trust it, the attacker can use that to provide a malicious script.
That's much harder to pull off though, because you need to replicate the environment close enough so that the victim doesn't suspect anything. Do they put their config files in /var/lib or random docker volumes? Do they use docker compose or docker-compose, etc.
It doesn't completely stop working; a MITM can still pretend to be the server, it just can't authenticate to the real server on your behalf. You could be doing all your work in a fake server controlled by the attacker, while the real server sits there untouched.
https://www.usenix.org/conference/srecon19emea/presentation/...
To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.
It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).
Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.
Or cat-ing some secrets that would be on target machine but not attacker