# HashiCorp Vault

The Vault is deployed as a small Ubuntu server (Proxmox VM) and is available at [https://vault.koszewscy.waw.pl](https://vault.koszewscy.waw.pl) from inside the Lazurowa network. This Vault instance is used as the primary source for storing secrets and is referred to as "Vault" or "main Vault" in the documentation.

The main Vault is automatically unsealed using Auto Unseal with the transit secrets engine in another Vault running on Proxmox Backup Server. The second Vault instance is referred to by the term "KMS Vault" in the documentation. It is available over HTTPS as [http://pbs.koszewscy.waw.pl:8200](http://pbs.koszewscy.waw.pl:8200) from inside the Lazurowa network.

Both instances are exposed to the network without using a reverse proxy. The TLS listener is bound to the private IP address of the host, and the HTTP listener is bound to the localhost interface only.

## Installation

```shell
curl -s https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=VERSION_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list
apt update && apt install vault
```

> **Note:** The original HashiCorp-supplied commands will fail on Debian/Proxmox because `UBUNTU_CODENAME` is not defined on those systems. The above command uses `VERSION_CODENAME` from `/etc/os-release` or falls back to `lsb_release -cs` (which must be installed).

## Development Mode

Vault can be started in development mode for testing purposes using the following (recommended) command:

```shell
vault server -dev -dev-no-store-token -dev-root-token-id="root-token" -dev-listen-address="127.0.0.1:8200"
```

- `-dev` - starts Vault in development mode
- `-dev-no-store-token` - prevents storing the root token on disk (`~/.vault-token`)
- `-dev-root-token-id="root-token"` - sets the root token to a known value for easy access
- `-dev-listen-address="127.0.0.1:8200"` - binds the Vault listener to localhost only, but can be changed to `0.0.0.0:8200` to allow external access

Then before running any Vault commands, set the following environment variables:

```shell
export VAULT_ADDR='http://127.0.0.1:8200' VAULT_TOKEN='root-token'
```

> **Warning:** Development mode is designed for experimentation only. Data is stored in-memory and will be lost when the Vault process is stopped.

## Local Mode

The repository also contains a set of scripts to run a local Vault instance. That instance may be used for experimentation, but unlike the development mode, it uses a file storage backend located at `./data`.

Start it using `bin/vault-start`. The script will create necessary directories and configuration files if they do not exist. The output from the `vault operator init` is saved to `config/vault-init.json`. The single unseal key and the root token are encrypted using GPG key available on the host system. The `bin/vault-unseal` script decrypts the unseal key and unseals the Vault. `set-env` script sets the `VAULT_TOKEN` and `VAULT_ADDR` environment variables. It should be sourced before running any Vault commands.

## TLS Certificate

Use standard Lego ACME installation instructions available at [Let's Encrypt directory](../letsencrypt/README.md) for both the main and the KMS Vault servers.

Although Proxmox Backup Server requests and provisions a public TLS certificate for the **pbs.koszewscy.waw.pl** domain using Let's Encrypt, the certificate file is stored in a location not accessible to Vault. Therefore, a separate TLS certificate must be created for the Vault server.

Create an executable certificate hook script for both Vault servers at `/usr/local/bin/certificate-hook`:

```shell
#!/usr/bin/bash

HOSTNAME=vault.koszewscy.waw.pl
install -o vault -g vault -m 0600 /etc/letsencrypt/certificates/${HOSTNAME}.crt /opt/vault/tls/tls.crt
install -o vault -g vault -m 0600 /etc/letsencrypt/certificates/${HOSTNAME}.key /opt/vault/tls/tls.key
systemctl restart vault
```

The script will be run after each successful certificate renewal. You will have to unseal the KMS Vault manually after each restart. You can add an email notification to the certificate renewal process to remind you about unsealing the KMS Vault.

## Auto Unseal Configuration

The default Vault unseal method uses Shamir's Secret Sharing, which requires manual entry of unseal keys after each Vault restart. To avoid this, the Auto Unseal feature is used, which leverages the transit secrets engine of another Vault instance (the KMS Vault).

### KMS Vault Setup

1. Enable the transit secrets engine:

    ```shell
    vault secrets enable transit
    ```

2. Create a new encryption key:

    ```shell
    vault write -f transit/keys/transit-unseal
    ```

3. Create a policy allowing unseal operations as a file named `transit-unseal-policy.hcl`:

    ```hcl
    path "transit/decrypt/unseal-key" {
        capabilities = ["update"]
    }

    path "transit/encrypt/unseal-key" {
        capabilities = ["update"]
    }
    ```

4. Apply the policy:

    ```shell
    vault policy write transit-unseal-policy transit-unseal-policy.hcl
    ```

5. Create a token with the policy attached:

    ```shell
    vault token create -policy="transit-unseal-policy"
    ```

    Save the generated token for later use.

6. Verify connectivity from the main Vault to the KMS Vault.

7. Store the KMS Vault unseal key and root token securely. Make an offline backup of the KMS Vault.

### Enabling audit logging (optional)

To enhance security and monitoring, enable audit logging on the KMS Vault:

```shell
mkdir -p /var/log/vault
chown -R vault:adm /var/log/vault
chmod 02750 /var/log/vault
vault audit enable file file_path="/var/log/vault/audit.log" mode="0640"
```

Then monitor the audit log online:

```shell
tail -F /var/log/vault/audit.log | jq -r '. | select(.type == "response") | [ .time, .request.path, .request.operation, .request.remote_address] | @tsv'
```

or offline:

```shell
jq -r '. | select(.type == "response") | [ .time, .request.path, .request.operation, .request.remote_address] | @tsv' /var/log/vault/audit.log | column -t -N "time,path,operation,remote_addr"
```

or using Alloy and Grafana. To use Alloy, add the following configuration to `config.alloy`:

```hcl
loki.source.file "vault_audit_log" {
    targets = [
        {"__path__" = "/var/log/vault/audit.log", "log_name" = "vault_audit"},
    ]
    forward_to = [loki.write.default.receiver]
}
```

Check the auditing configuration:

```shell
vault audit list -detailed
```

### Main Vault Configuration

Depending on main Vault state (new or existing), some of the following steps are mutually exclusive.

1. If the main Vault is already initialized, shut it down and back up its data directory and configuration file.

2. Update the main Vault configuration file (usually located at `/etc/vault.d/vault.hcl`) to include the Auto Unseal configuration:

    ```hcl
    seal "transit" {
      address            = "https://pbs.koszewscy.waw.pl:8200"
      key_name           = "transit-unseal"
      mount_path        = "transit/"
    }
    ```
3. Put the KMS Vault token created earlier into the environment file `/etc/vault.d/vault.env`:

    ```shell
    VAULT_TOKEN="s.xxxxxxx"
    ```

4. Ensure that the Vault service systemd unit file includes the environment file. If not, create an override file at `/etc/systemd/system/vault.service.d/override.conf`:

    ```ini
    [Service]
    EnvironmentFile=/etc/vault.d/vault.env
    ```

5. If the main Vault was already initialized, start it and unseal with `-migrate` parameter.

    ```shell
    systemctl start vault
    vault operator unseal -migrate
    ```

6. Verify that the main Vault is unsealed and operational. The Seal Type should show `transit`, and Recovery Seal Type should show `shamir`.

7. Uninitialized main Vault will automatically encrypt the root key with the transit key from the KMS Vault during initialization and present recovery keys for Shamir's Secret Sharing.

## Offline Backup

Vault installs the following directories:

- `/etc/vault.d` - configuration files
- `/opt/vault/data` - default file storage backend
- `/opt/vault/tls` - TLS certificates

The simplest method of backing up Vault is to stop the Vault service, back up the above directories, and then start the service again. This can be automated using systemd service and timer units.

Systemd service unit `/etc/systemd/system/backup-vault.service`:

```ini
[Unit]
Description=Make an off-line backup of HashiCorp Vault

[Service]
EnvironmentFile=/etc/proxmox-backup/pbc-vault.env
ExecStartPre=/usr/bin/systemctl stop vault.service
ExecStart=/usr/bin/proxmox-backup-client backup --backup-id "vault" --backup-type host config.pxar:/etc/vault.d data.pxar:/opt/vault/data tls.pxar:/opt/vault/tls
ExecStartPost=/usr/bin/systemctl start vault.service
RemainAfterExit=no
Type=oneshot
```

Systemd timer unit `/etc/systemd/system/backup-vault.timer`:

```ini
[Unit]
Description=Make an off-line backup of HashiCorp Vault (timer)

[Timer]
Persistent=true
OnCalendar=*-*-* 2:40
RandomizedDelaySec=1h

[Install]
WantedBy=timers.target
```

The KMS Vault is protected by regular unseal key(s) and therefore cannot be backed up automatically in the same way. Use the script below to create a backup of the KMS Vault and start it up afterwards.

```shell
#!/usr/bin/bash

PBC_ENV="/etc/proxmox-backup/pbc-vault.env"
if [[ -f $PBC_ENV ]]; then
    # Import variables and export them for the proxmox-backup-client usage
    export $(grep -v ^# $PBC_ENV)
fi

# Stop Vault service
systemctl stop vault

# Create a backup
proxmox-backup-client backup --backup-id "vault-kms" --backup-type host confg.pxar:/etc/vault.d data.pxar:/opt/vault/data tls.pxar:/opt/vault/tls

# Start Vault service
systemctl start vault

# Unseal the Vault
VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal
```

## 1Password Setup

Install 1Password CLI using the commands below:

```shell
VERSION="v2.32.0" # I don't know how to get the latest version dynamically
# Repleace aarch with arm64
ARCH=${ARCH/aarch64/arm64}
echo "Installing 1Password CLI version ${VERSION} manually..."
TMP="/tmp/$(mktemp -d 1password-cli-installtion.XXXXXX)"
mkdir -p $TMP/extracted
curl -vsSL "https://cache.agilebits.com/dist/1P/op2/pkg/${VERSION}/op_linux_${ARCH}_${VERSION}.zip" -o $TMP/op.zip
unzip $TMP/op.zip -d $TMP/extracted
sudo groupadd -f onepassword-cli
sudo install -g onepassword-cli $TMP/extracted/op /usr/local/bin/op
sudo chmod g+s /usr/local/bin/op
if [[ -d $TMP ]]; then rm -rf $TMP; fi
```

> **Note:** Adjust the `VERSION` variable to install a different version of the 1Password CLI.

Create a `.vault.env` file in the home directory with the following content:

```shell
VAULT_ADDR=http://localhost:8200
VAULT_TOKEN=op://Private/root KMS Koszewscy/password
```

add the following line to your shell profile (e.g., `~/.bashrc` or `~/.zshrc`):

```shell
alias op_vault='op run --env-file="$HOME/.vault.env" -- vault'
```

Run Vault CLI as follows:

```shell
op_vault token lookup
```

Eventually, set both `VAULT_ADDR` and `VAULT_TOKEN` environment variables in your shell:

```shell
export $(cat $HOME/.vault.env | op inject)
```