The Vault is deployed as a small Ubuntu server (Proxmox VM) and is available at https://vault.koszewscy.waw.pl from inside the Lazurowa network. This Vault instance is used as the primary source for storing secrets and is referred to as "Vault" or "main Vault" in the documentation.

The main Vault is automatically unsealed using Auto Unseal with the transit secrets engine in another Vault running on Proxmox Backup Server. The second Vault instance is referred to by the term "KMS Vault" in the documentation. It is available over HTTPS as http://pbs.koszewscy.waw.pl:8200 from inside the Lazurowa network.

Both instances are exposed to the network without using a reverse proxy. The TLS listener is bound to the private IP address of the host, and the HTTP listener is bound to the localhost interface only.

Installation

curl -s https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=VERSION_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list
apt update && apt install vault

Note: The original HashiCorp-supplied commands will fail on Debian/Proxmox because UBUNTU_CODENAME is not defined on those systems. The above command uses VERSION_CODENAME from /etc/os-release or falls back to lsb_release -cs (which must be installed).

Development Mode

Vault can be started in development mode for testing purposes using the following (recommended) command:

vault server -dev -dev-no-store-token -dev-root-token-id="root-token" -dev-listen-address="127.0.0.1:8200"

-dev - starts Vault in development mode
-dev-no-store-token - prevents storing the root token on disk (~/.vault-token)
-dev-root-token-id="root-token" - sets the root token to a known value for easy access
-dev-listen-address="127.0.0.1:8200" - binds the Vault listener to localhost only, but can be changed to 0.0.0.0:8200 to allow external access

Then before running any Vault commands, set the following environment variables:

export VAULT_ADDR='http://127.0.0.1:8200' VAULT_TOKEN='root-token'

Warning: Development mode is designed for experimentation only. Data is stored in-memory and will be lost when the Vault process is stopped.

Local Mode

The repository also contains a set of scripts to run a local Vault instance. That instance may be used for experimentation, but unlike the development mode, it uses a file storage backend located at ./data.

Start it using bin/vault-start. The script will create necessary directories and configuration files if they do not exist. The output from the vault operator init is saved to config/vault-init.json. The single unseal key and the root token are encrypted using GPG key available on the host system. The bin/vault-unseal script decrypts the unseal key and unseals the Vault. set-env script sets the VAULT_TOKEN and VAULT_ADDR environment variables. It should be sourced before running any Vault commands.

TLS Certificate

Use standard Lego ACME installation instructions available at Let's Encrypt directory for both the main and the KMS Vault servers.

Although Proxmox Backup Server requests and provisions a public TLS certificate for the pbs.koszewscy.waw.pl domain using Let's Encrypt, the certificate file is stored in a location not accessible to Vault. Therefore, a separate TLS certificate must be created for the Vault server.

Create an executable certificate hook script for both Vault servers at /usr/local/bin/certificate-hook:

#!/usr/bin/bash

HOSTNAME=vault.koszewscy.waw.pl
install -o vault -g vault -m 0600 /etc/letsencrypt/certificates/${HOSTNAME}.crt /opt/vault/tls/tls.crt
install -o vault -g vault -m 0600 /etc/letsencrypt/certificates/${HOSTNAME}.key /opt/vault/tls/tls.key
systemctl restart vault

The script will be run after each successful certificate renewal. You will have to unseal the KMS Vault manually after each restart. You can add an email notification to the certificate renewal process to remind you about unsealing the KMS Vault.

Auto Unseal Configuration

The default Vault unseal method uses Shamir's Secret Sharing, which requires manual entry of unseal keys after each Vault restart. To avoid this, the Auto Unseal feature is used, which leverages the transit secrets engine of another Vault instance (the KMS Vault).

KMS Vault Setup

Enable the transit secrets engine:
```
vault secrets enable transit
```

Create a new encryption key:

vault write -f transit/keys/transit-unseal

Create a policy allowing unseal operations as a file named transit-unseal-policy.hcl:

path "transit/decrypt/unseal-key" {
    capabilities = ["update"]
}

path "transit/encrypt/unseal-key" {
    capabilities = ["update"]
}

Apply the policy:

vault policy write transit-unseal transit_unseal_policy.hcl

Create a token with the policy attached:

vault token create -orphan -policy="transit-unseal" -wrap-ttl=120 -period=24h -field=wrapping_token > wrapping-token.txt

Copy the wrapping-token.txt file securely to the main Vault server. It can be copied using scp or any other secure method.
Store the KMS Vault unseal key and root token securely. Make an offline backup of the KMS Vault.

Enabling audit logging (optional)

To enhance security and monitoring, enable audit logging on the KMS Vault:

mkdir -p /var/log/vault
chown -R vault:adm /var/log/vault
chmod 02750 /var/log/vault
vault audit enable file file_path="/var/log/vault/audit.log" mode="0640"

Then monitor the audit log online:

tail -F /var/log/vault/audit.log | jq -r '. | select(.type == "response") | [ .time, .request.path, .request.operation, .request.remote_address] | @tsv'

or offline:

jq -r '. | select(.type == "response") | [ .time, .request.path, .request.operation, .request.remote_address] | @tsv' /var/log/vault/audit.log | column -t -N "time,path,operation,remote_addr"

or using Alloy and Grafana. To use Alloy, add the following configuration to config.alloy:

loki.source.file "vault_audit_log" {
    targets = [
        {"__path__" = "/var/log/vault/audit.log", "log_name" = "vault_audit"},
    ]
    forward_to = [loki.write.default.receiver]
}

Check the auditing configuration:

vault audit list -detailed

Main Vault Configuration

Depending on main Vault state (new or existing), some of the following steps are mutually exclusive.

If the main Vault is already initialized, shut it down and back up its data directory and configuration file.

Verify connectivity from the main Vault to the KMS Vault.

VAULT_ADDR=https://kms.koszewscy.waw.pl:8200 vault status

curl -s https://kms.koszewscy.waw.pl:8200/v1/sys/seal-status | jq .

Update the main Vault configuration file (usually located at /etc/vault.d/vault.hcl) to include the Auto Unseal configuration:

ui = true

storage "file" {
  path = "/opt/vault/data"
}

# HTTP listener
listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = "true"
}

# HTTPS listener
listener "tcp" {
  address       = "192.168.2.10:443"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
}

seal "transit" {
  address         = "https://kms.koszewscy.waw.pl:8200"
  disable_renewal = "false"
  key_name        = "unseal-key"
  mount_path      = "transit/"
}

Unwrap the token to get the KMS Vault token:

VAULT_ADDR=https://kms.koszewscy.waw.pl:8200 vault unwrap -field=token $(cat wrapping-token.txt) > kms-vault-token.txt

Put the unwrapped KMS Vault token into the environment file /etc/vault.d/vault.env:
```
VAULT_TOKEN="s.xxxxxxx"
```
Ensure that the Vault service systemd unit file includes the environment file. If not, create an override file at /etc/systemd/system/vault.service.d/override.conf:
```
[Service]
EnvironmentFile=/etc/vault.d/vault.env
```
Add systemd override /etc/systemd/system/vault.service.d/override.conf if not already present:
```
[Service]
AmbientCapabilities=CAP_IPC_LOCK CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
```
That will allow the vault process to bind to low-numbered ports (443) and lock memory.
If the main Vault was already initialized, start it and unseal with -migrate parameter.
```
systemctl start vault
vault operator unseal -migrate
```
Verify that the main Vault is unsealed and operational. The Seal Type should show transit, and Recovery Seal Type should show shamir.
Uninitialized main Vault will automatically encrypt the root key with the transit key from the KMS Vault during initialization and present recovery keys for Shamir's Secret Sharing.

If for any reason the Auto Unseal method fails, you can always unseal the main Vault using the recovery keys provided during initialization. If the token expires, generate a new one using the KMS Vault, and transfer it to the main Vault as described above.

Offline Backup

Vault installs the following directories:

/etc/vault.d - configuration files
/opt/vault/data - default file storage backend
/opt/vault/tls - TLS certificates

The simplest method of backing up Vault is to stop the Vault service, back up the above directories, and then start the service again. This can be automated using systemd service and timer units.

Systemd service unit /etc/systemd/system/backup-vault.service:

[Unit]
Description=Make an off-line backup of HashiCorp Vault

[Service]
EnvironmentFile=/etc/proxmox-backup/pbc-vault.env
ExecStartPre=/usr/bin/systemctl stop vault.service
ExecStart=/usr/bin/proxmox-backup-client backup --backup-id "vault" --backup-type host config.pxar:/etc/vault.d data.pxar:/opt/vault/data tls.pxar:/opt/vault/tls
ExecStartPost=/usr/bin/systemctl start vault.service
RemainAfterExit=no
Type=oneshot

Systemd timer unit /etc/systemd/system/backup-vault.timer:

[Unit]
Description=Make an off-line backup of HashiCorp Vault (timer)

[Timer]
Persistent=true
OnCalendar=*-*-* 2:40
RandomizedDelaySec=1h

[Install]
WantedBy=timers.target

The KMS Vault is protected by regular unseal key(s) and therefore cannot be backed up automatically in the same way. Use the script below to create a backup of the KMS Vault and start it up afterwards.

#!/usr/bin/bash

PBC_ENV="/etc/proxmox-backup/pbc-vault.env"
if [[ -f $PBC_ENV ]]; then
    # Import variables and export them for the proxmox-backup-client usage
    export $(grep -v ^# $PBC_ENV)
fi

# Stop Vault service
systemctl stop vault

# Create a backup
proxmox-backup-client backup --backup-id "vault-kms" --backup-type host confg.pxar:/etc/vault.d data.pxar:/opt/vault/data tls.pxar:/opt/vault/tls

# Start Vault service
systemctl start vault

# Unseal the Vault
VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal

1Password Setup

Install 1Password CLI using the commands below:

VERSION="v2.32.0" # I don't know how to get the latest version dynamically
# Repleace aarch with arm64
ARCH=${ARCH/aarch64/arm64}
echo "Installing 1Password CLI version ${VERSION} manually..."
TMP="/tmp/$(mktemp -d 1password-cli-installtion.XXXXXX)"
mkdir -p $TMP/extracted
curl -vsSL "https://cache.agilebits.com/dist/1P/op2/pkg/${VERSION}/op_linux_${ARCH}_${VERSION}.zip" -o $TMP/op.zip
unzip $TMP/op.zip -d $TMP/extracted
sudo groupadd -f onepassword-cli
sudo install -g onepassword-cli $TMP/extracted/op /usr/local/bin/op
sudo chmod g+s /usr/local/bin/op
if [[ -d $TMP ]]; then rm -rf $TMP; fi

Note: Adjust the VERSION variable to install a different version of the 1Password CLI.

Create a .vault.env file in the home directory with the following content:

VAULT_ADDR=http://localhost:8200
VAULT_TOKEN=op://Private/root KMS Koszewscy/password

add the following line to your shell profile (e.g., ~/.bashrc or ~/.zshrc):

alias op_vault='op run --env-file="$HOME/.vault.env" -- vault'

Run Vault CLI as follows:

op_vault token lookup

Eventually, set both VAULT_ADDR and VAULT_TOKEN environment variables in your shell:

export $(cat $HOME/.vault.env | op inject)