Post-Quantum SSH on VPS: A Comprehensive Guide

Before modifying your SSH configuration, ensure your VPS can build and run a post-quantum-capable OpenSSH stack. This guide assumes you have a modern Linux distribution with a working package manager, root access, and a recovery plan such as a second SSH session or out-of-band console access).

For additional precautions, review our guide on SSH key security best practices to further secure your environment.

Prerequisites Checklist

• OS: Debian 11/12, Ubuntu 22.04/24.04, Rocky Linux 8/9, or CentOS Stream 8/9
• Package manager: `apt`, `dnf`, or `yum`
• Access: `root` shell or a sudo-capable admin user
• Required packages: compiler toolchain, `git`, `cmake`, OpenSSL development headers, and zlib development headers
• Network: Outbound HTTPS access for fetching source code and dependencies
• Safety net: A backup SSH session or out-of-band console access in case `sshd` fails to restart

If your VPS can compile software and you have a recovery plan, you can experiment with post-quantum SSH without risking lockout. This method helps mitigate a potential “harvest now, decrypt later” attack where adversaries capture SSH traffic today to decrypt in the future.

For more on securing your VPS, check our post on VPS security practices.

Install OpenSSH and Liboqs Library

The goal is to build a functioning SSH stack on your server and integrate the liboqs cryptographic library, which provides access to post-quantum algorithms. Standard distro packages on Debian 11 or CentOS 8 do not include full post-quantum key exchange support, so compiling liboqs from source is necessary.

On both distributions, you need two layers of packages:

1. OpenSSH packages: Provides the SSH client, server, and baseline configuration files.
2. Build dependencies: Needed to compile liboqs and the patched OpenSSH variant.

Debian 11

Install OpenSSH and the necessary build tools and headers:

sudo apt update
sudo apt install -y \
  openssh-client openssh-server \
  build-essential git cmake ninja-build pkg-config \
  libssl-dev zlib1g-dev libpam0g-dev

The installation of `openssh-server` provides `sshd`, while `libssl-dev` and `zlib1g-dev` supply the required crypto and compression headers.

Compile and install liboqs from source:

git clone --depth 1 https://github.com/open-quantum-safe/liboqs.git
cd liboqs
cmake -S . -B build -GNinja -DCMAKE_BUILD_TYPE=Release
cmake --build build
sudo cmake --install build
sudo ldconfig

Verify the installation:

ldconfig -p | grep liboqs

Expected output should include a line similar to:

liboqs.so.0 => /usr/local/lib/liboqs.so.0

CentOS 8

For CentOS 8, first install the development tools group, OpenSSH, and the necessary development headers:

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y epel-release
sudo dnf install -y \
  openssh-clients openssh-server \
  git cmake ninja-build pkgconfig \
  openssl-devel zlib-devel pam-devel

Here, the `Development Tools` group provides GCC, `make`, and similar tools; `openssl-devel` and `zlib-devel` are mandatory for compiling a working OpenSSH binary.

Then, build and install `liboqs` in the same manner:

git clone --depth 1 https://github.com/open-quantum-safe/liboqs.git
cd liboqs
cmake -S . -B build -GNinja -DCMAKE_BUILD_TYPE=Release
cmake --build build
sudo cmake --install build
sudo ldconfig

`liboqs` is the engine room for post-quantum SSH. It implements primitives like **ML-KEM/Kyber-style key encapsulation**, designed to resist quantum attacks. Think of OpenSSH as the protocol driver and `liboqs` as the enhanced cryptographic engine.

Compile and Patch OpenSSH for PQ Key Exchange

In this step, you add post-quantum key exchange capabilities to your SSH stack. With liboqs installed, obtain an OpenSSH source tree suited for integrating this library. Use the Open Quantum Safe patch set for a matching OpenSSH release. For example:

• OpenSSH source: `openssh-9.6p1.tar.gz`
• Patch tree: the `oqs-openssh` branch targeting 9.6p1

Download and prepare the sources:

curl -LO https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.6p1.tar.gz
tar -xf openssh-9.6p1.tar.gz
git clone --depth 1 https://github.com/open-quantum-safe/openssh.git oqs-openssh
grep -R "9.6p1" oqs-openssh 2>/dev/null | head

Verify that both source directories exist:

test -d openssh-9.6p1 && test -d oqs-openssh && echo "sources present"

Expected output:

sources present

A minimal patch touches three areas: algorithm registration, build system integration, and linker flags. For example:

diff --git a/kex-names.c b/kex-names.c
index 1111111..2222222 100644
--- a/kex-names.c
+++ b/kex-names.c
@@ -60,6 +60,10 @@ static const char *kexalgs[] = {
        "curve25519-sha256",
        "curve25519-sha256@libssh.org",
        "sntrup761x25519-sha512@openssh.com",
+       "mlkem768x25519-sha256",
+       "mlkem1024nistp384-sha384",
+       "frodo640aesx25519-sha256",
+       "kyber768x25519-sha256",
        NULL
 };

This patch registers new hybrid key exchange methods so that SSH negotiation can select a post-quantum handshake if available.

Configure sshd_config for Post-Quantum Algorithms

After compiling the PQ-capable OpenSSH, configure `sshd` to prefer these algorithms. A recommended strategy is to list hybrid post-quantum key exchanges first, with traditional methods as fallback. Before editing, confirm which key exchange identifiers your build supports:

/usr/sbin/sshd -T | tr ' ' '\n' | grep '^kexalgorithms' -A1
ssh -Q kex | grep -E 'mlkem|kyber|frodo|sntrup|curve25519'

Once confirmed, update the relevant line in `/etc/ssh/sshd_config`. For example:

diff
--- /etc/ssh/sshd_config.orig
+++ /etc/ssh/sshd_config
@@
-#KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,sntrup761x25519-sha512@openssh.com
+KexAlgorithms mlkem768x25519-sha256,kyber768x25519-sha256,sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256

Validate the syntax before proceeding:

sudo sshd -t -f /etc/ssh/sshd_config

A return code of `0` confirms a valid configuration.

Restart SSH and Verify the PQ Handshake

Reload or restart the SSH service without interrupting your current session. On systemd systems, use:

sudo systemctl reload sshd || sudo systemctl restart sshd

On Debian/Ubuntu where the service may be named `ssh`:

sudo systemctl reload ssh || sudo systemctl restart ssh

For SysV init systems:

sudo service ssh reload || sudo service ssh restart
# or
sudo service sshd reload || sudo service sshd restart

Verify the service status:

sudo systemctl status sshd --no-pager -l 2>/dev/null || \
sudo systemctl status ssh --no-pager -l 2>/dev/null || \
service sshd status 2>/dev/null || \
service ssh status

Open a new terminal and test a connection with verbose output to confirm the negotiated PQ algorithm:

ssh -vv \
-o KexAlgorithms=mlkem768x25519-sha256,kyber768x25519-sha256,sntrup761x25519-sha512@openssh.com \
admin@your-server-ip

Look for an output line indicating a PQ algorithm, such as:

debug1: kex: algorithm: mlkem768x25519-sha256

For additional security insights, review our article on VPS security questions answered.

Measure Handshake Performance on Your VPS

After verifying the configuration, benchmark the handshake performance. Key metrics include handshake time (from connection initiation to authentication prompt) and CPU usage. In a typical use case, an extra few milliseconds is acceptable, but in high-load environments like CI/CD systems or automation tasks, these metrics can add up.

If you’re setting up your post-quantum SSH on a VPS, consider using VPSus KVM VPS Hosting. For instance, our KVM1 plan starts at $10/mo and provides 1 vCore, 1 GB ECC RAM, and NVMe storage in regions such as Atlanta and Frankfurt, making it an ideal low-cost option for experimental setups.

On a well-provisioned VPS, a hybrid PQ handshake should take under 5 ms on low-latency routes. Use the benchmarking script below to compare classical and post-quantum handshakes:

benchmark_placeholder

After running the benchmark, check the log:

tail -n 5 pq-ssh-benchmark.log

Example log output:

run=16 handshake_ms=4.3 cpu_pct=12.1 kex=mlkem768x25519-sha256
run=17 handshake_ms=4.1 cpu_pct=11.8 kex=mlkem768x25519-sha256
run=18 handshake_ms=4.5 cpu_pct=12.4 kex=mlkem768x25519-sha256

Automate PQ SSH Deployment with Ansible

Once your benchmark meets expectations, scale the deployment using Ansible to distribute the new OpenSSH build and configuration across multiple VPS nodes. Ansible ensures idempotent deployments that avoid unnecessary changes. An example inventory file might look like:

ini
# inventory.ini
[pq_ssh]
edge-01 ansible_host=203.0.113.10 ansible_user=admin
edge-02 ansible_host=203.0.113.11 ansible_user=admin
db-01   ansible_host=203.0.113.12 ansible_user=admin

[pq_ssh:vars]
ansible_become=true
ansible_python_interpreter=/usr/bin/python3

And a sample playbook for deployment:

- name: Deploy post-quantum OpenSSH
  hosts: pq_ssh
  become: true
  vars:
    ssh_service_name: "{{ 'ssh' if ansible_os_family == 'Debian' else 'sshd' }}"
    pq_kex: "mlkem768x25519-sha256,kyber768x25519-sha256,sntrup761x25519-sha512@openssh.com,curve25519-sha256"
  tasks:
    - name: Install runtime packages on Debian
      ansible.builtin.apt:
        name:
          - libssl3
          - zlib1g
          - libpam0g
        state: present
        update_cache: true
      when: ansible_os_family == "Debian"

    - name: Install runtime packages on RedHat
      ansible.builtin.package:
        name:
          - openssl-libs
          - zlib
          - pam
        state: present
      when: ansible_os_family == "RedHat"

    - name: Copy patched ssh client binary
      ansible.builtin.copy:
        src: files/usr_bin_ssh
        dest: /usr/bin/ssh
        owner: root
        group: root
        mode: '0755'
        backup: true
      notify: restart ssh service

    - name: Copy patched sshd server binary
      ansible.builtin.copy:
        src: files/usr_sbin_sshd
        dest: /usr/sbin/sshd
        owner: root
        group: root
        mode: '0755'
        backup: true
      notify: restart ssh service

    - name: Set PQ KEX in sshd_config
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^KexAlgorithms '
        line: "KexAlgorithms {{ pq_kex }}"
        create: false
        backup: true
      notify: validate and restart ssh service

    - name: Ensure KexAlgorithms exists if missing
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        line: "KexAlgorithms {{ pq_kex }}"
        insertafter: EOF
      when: ansible_facts is defined
      notify: validate and restart ssh service

  handlers:
    - name: restart ssh service
      ansible.builtin.service:
        name: "{{ ssh_service_name }}"
        state: restarted

    - name: validate and restart ssh service
      ansible.builtin.command: /usr/sbin/sshd -t -f /etc/ssh/sshd_config
      register: sshd_test
      changed_when: false
      listen: validate and restart ssh service

    - name: restart validated ssh service
      ansible.builtin.service:
        name: "{{ ssh_service_name }}"
        state: restarted
      when: sshd_test.rc == 0
      listen: validate and restart ssh service

This playbook automates the secure deployment of your post-quantum SSH configuration across your VPS fleet.