Skip to content

Architecture

Vault and service handshake

The vault stores signing keys, service verifier keys, the admin verification key, per-volume encryption keys, the API-key master secret, and the credentials for the object backends behind each storage. The vault is required, not optional. Every service-to-service handshake reads from it.

Two vaults per deployment

mountOS uses two logical vaults: a Hub vault and a Region vault per region. The Hub vault stands alongside the HUB. The two vaults are independent. mountOS does not write across them. Replicate the shared pieces separately. mountOS does not automate this replication.

Hub vault

The Hub vault exists once per deployment, alongside the HUB. It holds the HUB's own secrets and the authoritative copy of the service-verifier public keys.

  • appserv. The HUB's ED25519 signing and verification keys, the admin verification key, which appserv uses to verify Admin SDK JWTs, and the admin database connection details.
  • service-verifiers. One public key per service (appserv, dataserv, gcserv, blockserv). This path is the source of truth for the whole deployment.

Region vault

One Region vault exists per region, alongside dataserv. It holds the regional services' own secrets and a replicated copy of the service-verifier set. Regional handshakes never need to call back to the HUB.

A single Region vault serves all metadata clusters in that region. Clustering is a volume-load pool inside the region. It does not partition the vault or the database. Every metadata cluster in a region shares that region's one vault and one database.

  • dataserv, gcserv, blockserv. Each service's ED25519 signing and verification keys. dataserv and gcserv also keep the data database connection here.
  • service-verifiers. A replica of the Hub vault's verifier set. The operator keeps this replica in sync with a replication policy outside the Admin API.
  • s3creds. Credentials for the object backends behind the region's storages (AWS, Backblaze, MinIO, and so on). dataserv or gcserv creates and rotates these credentials.
  • volcreds. Per-volume encryption keys. Write-once at the application level, created on first use. gcserv deletes the entry once the volume is deactivated, vault-cleanup is enabled, and the grace period passes.
  • api-master. The master secret that protects stored API keys. Independent per region. The operator rotates it on a schedule. gcserv re-encrypts stored API keys to the latest version.

The service handshake

Every service-to-service call carries a signed JWT. The receiver verifies the JWT with the sender's public key from its local service-verifiers/ path. Rotation is zero-downtime. Regional services never have to call the HUB to verify a JWT. The replicated verifier set in their own Region vault is enough.

The vault stores keys as raw base64 bytes.

Expected fields

Every vault path holds a fixed schema. Seed the service secrets at install time. The application creates s3creds and volcreds on first use. Write api-master versions as part of the API-key rotation policy. Download each template below as a starting point. Paths below show the default mountos root. An optional RESOURCE_PREFIX env var renames it to mountos-<prefix> across all four backends. Multiple deployments can then share one cloud account without secret name collisions.

appserv (Hub vault)

mountos/appserv env download
# mountOS appserv vault secrets
# Path: mountos/appserv  (Hub vault, one per deployment)
#
# Seed once at install time. Rotate the ED25519 key pair on the
# schedule defined in the rotation policy (typically weekly/monthly).
#
# All ED25519 keys are raw bytes, standard base64 (not URL-safe).
#   Signing key:      64 bytes -> 88 base64 chars
#   Verification key: 32 bytes -> 44 base64 chars
# SPKI, PEM, and PKCS8 wrappers are rejected by the verifier.

# Admin database connection
DB_DIALECT=postgresql          # mysql | postgresql
DB_URL=                        # full connection URL for the admin database
DB_PROVIDER=postgresql         # mysql | postgresql | tidb | ...
DB_PROVIDER_VERSION=           # optional, hint for capability detection (e.g. 15.2, 8.0.32)

# HUB (appserv) signing key pair
ED25519_SIGNING_KEY=           # 88 base64 chars
ED25519_VERIFICATION_KEY=      # 44 base64 chars

# Admin SDK verification key (operator ED25519 public key)
PROVIDER_VERIFICATION_KEY=     # 44 base64 chars, operator ED25519 public key

# Dashboard-user header HMAC secret. Dedicated random value (>= 32 bytes),
# distinct from PROVIDER_VERIFICATION_KEY (which is public). Signs the
# X-MountOS-Dashboard-User header; share only with the dashboard backend.
DASHBOARD_USER_HMAC_KEY=       # random secret, >= 32 bytes

dataserv (Region vault)

mountos/dataserv env download
# mountOS dataserv vault secrets
# Path: mountos/dataserv  (Region vault, one per region)
#
# Seed once at install time in every region that runs dataserv.
# Rotate the ED25519 key pair per the rotation policy.
#
# All ED25519 keys are raw bytes, standard base64.
#   Signing key:      64 bytes -> 88 base64 chars
#   Verification key: 32 bytes -> 44 base64 chars
# SPKI, PEM, and PKCS8 wrappers are rejected.

# Regional data database connection
DB_DIALECT=postgresql          # mysql | postgresql
DB_URL=                        # full connection URL for the data database
DB_PROVIDER=postgresql         # mysql | postgresql | tidb | ...
DB_PROVIDER_VERSION=           # optional, hint for capability detection

# dataserv signing key pair
ED25519_SIGNING_KEY=           # 88 base64 chars
ED25519_VERIFICATION_KEY=      # 44 base64 chars

gcserv (Region vault)

mountos/gcserv env download
# mountOS gcserv vault secrets
# Path: mountos/gcserv  (Region vault, one per region)
#
# Seed once at install time in every region that runs gcserv.
# Rotate the ED25519 key pair per the rotation policy.
#
# All ED25519 keys are raw bytes, standard base64.
#   Signing key:      64 bytes -> 88 base64 chars
#   Verification key: 32 bytes -> 44 base64 chars
# SPKI, PEM, and PKCS8 wrappers are rejected.

# Regional data database connection
DB_DIALECT=postgresql          # mysql | postgresql
DB_URL=                        # full connection URL for the data database
DB_PROVIDER=postgresql         # mysql | postgresql | tidb | ...
DB_PROVIDER_VERSION=           # optional, hint for capability detection

# gcserv signing key pair
ED25519_SIGNING_KEY=           # 88 base64 chars
ED25519_VERIFICATION_KEY=      # 44 base64 chars

blockserv (Region vault)

mountos/blockserv env download
# mountOS blockserv vault secrets
# Path: mountos/blockserv  (Region vault, one per region)
#
# Seed once at install time. Rotate per rotation policy.
# ED25519 keys are raw bytes, standard base64.

ED25519_SIGNING_KEY=           # 88 base64 chars (64 raw bytes)
ED25519_VERIFICATION_KEY=      # 44 base64 chars (32 raw bytes)

service-verifiers/<service>

mountos/service-verifiers/<service> env download
# mountOS service-verifier entry
# Path: mountos/service-verifiers/<service>
#   where <service> is one of:
#     appserv, dataserv, gcserv, blockserv
#
# The Hub vault holds the source of truth for every service's
# ED25519 public key. Replicate this path to every
# Region vault so cross-service JWT verification never has to
# call back to the HUB.
#
# Create one entry per service. Value is the matching public key
# stored in mountos/<service>/ED25519_VERIFICATION_KEY.

ED25519_VERIFICATION_KEY=      # 44 base64 chars (32 raw bytes)

api-master (Region vault)

mountos/api-master env download
# mountOS api-master entry
# Path: mountos/api-master  (Region vault, one per region)
#
# A single random string used to derive the 32-byte master key that
# encrypts API-key ciphertexts in the regional admin database.
# Independent per region.
#
# Rotation: write a new version. gcserv picks up the
# latest version via its MasterKeyAlignmentGoal and re-encrypts
# stored ciphertexts under the new master key. Services cache the
# one or two most recent versions.

key=                           # random string, at least 12 characters

s3creds/<bucket>, volcreds/<volume>

The application manages these paths. dataserv or gcserv writes these on first use. Grant the matrix-defined permissions on these paths. Do not seed the values.

Vault providers

Every service selects its vault with VAULT_PROVIDER plus the provider-specific credential variables. The operator can mix vault providers across regions. The operator configures each vault independently.

  • Configure one vault per region plus the Hub vault near the HUB.
  • Grant the matrix-defined permissions to each service.
  • Replicate service-verifiers/ from the Hub vault into every region.

Each setup script below also accepts the same RESOURCE_PREFIX env var and applies it to every name it creates.

HashiCorp Vault

The setup script mounts the KVv2 engine at mountos/. It uses AppRole auth with automatic token renewal.

Service env vars:

  • VAULT_PROVIDER=hashicorp
  • VAULT_HASHICORP_ADDRESS
  • VAULT_HASHICORP_ROLE_ID, VAULT_HASHICORP_SECRET_ID
hashicorp-vault-setup.sh bash download
#!/usr/bin/env bash
# mountOS HashiCorp Vault setup
#
# Run the hub-vault block once on the vault near the HUB, and the
# region-vault block once per Region vault instance.
#
# Prereqs:
#   - VAULT_ADDR + VAULT_TOKEN exported (root/management token for setup)
#   - vault CLI available
#
# Policies and AppRoles defined here match the permission matrix on
# the Vault and service handshake page.

set -euo pipefail

RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

# ------------------------------------------------------------
# Hub Vault (deployed alongside the HUB, one per deployment)
# ------------------------------------------------------------

vault auth enable approle 2>/dev/null || true
vault secrets enable -path=$NAME_ROOT kv-v2 2>/dev/null || true

# appserv: read own secrets + full write on the service-verifier set
cat > appserv-policy.hcl <<EOF
path "$NAME_ROOT/data/appserv" {
  capabilities = ["read"]
}
path "$NAME_ROOT/data/service-verifiers/*" {
  capabilities = ["create", "read", "update", "list"]
}
EOF
vault policy write appserv-policy appserv-policy.hcl
vault write auth/approle/role/appserv-role token_policies="appserv-policy"

# ------------------------------------------------------------
# Region Vault (one per region, alongside the dataserv metadata cluster)
# ------------------------------------------------------------

vault auth enable approle 2>/dev/null || true
vault secrets enable -path=$NAME_ROOT kv-v2 2>/dev/null || true

# Base policy per service: read own + replicated service-verifiers + api-master
for svc in dataserv gcserv blockserv; do
cat > ${svc}-policy.hcl <<EOF
path "$NAME_ROOT/data/${svc}" {
  capabilities = ["read"]
}
path "$NAME_ROOT/data/service-verifiers/*" {
  capabilities = ["read"]
}
path "$NAME_ROOT/data/api-master" {
  capabilities = ["read"]
}
path "$NAME_ROOT/metadata/api-master" {
  capabilities = ["read"]
}
EOF
  vault policy write ${svc}-policy ${svc}-policy.hcl
done

# s3creds: dataserv/gcserv full CRUD, blockserv read-only
cat > s3creds-full.hcl <<EOF
path "$NAME_ROOT/data/s3creds/*" {
  capabilities = ["create", "update", "read", "delete", "list"]
}
EOF
vault policy write s3creds-full s3creds-full.hcl

cat > s3creds-readonly.hcl <<EOF
path "$NAME_ROOT/data/s3creds/*" {
  capabilities = ["read", "list"]
}
EOF
vault policy write s3creds-readonly s3creds-readonly.hcl

# volcreds: dataserv create+read (write-once, no delete)
cat > volcreds-create.hcl <<EOF
path "$NAME_ROOT/data/volcreds/*" {
  capabilities = ["create", "read", "list"]
}
EOF
vault policy write volcreds-create volcreds-create.hcl

# volcreds: gcserv create+read+delete (post-deactivation cleanup)
cat > volcreds-full.hcl <<EOF
path "$NAME_ROOT/data/volcreds/*" {
  capabilities = ["create", "read", "delete", "list"]
}
EOF
vault policy write volcreds-full volcreds-full.hcl

# AppRole bindings per the matrix
vault write auth/approle/role/dataserv-role \
  token_policies="dataserv-policy,s3creds-full,volcreds-create"

vault write auth/approle/role/gcserv-role \
  token_policies="gcserv-policy,s3creds-full,volcreds-full"

vault write auth/approle/role/blockserv-role \
  token_policies="blockserv-policy,s3creds-readonly"

# Fetch role-id / secret-id per service and inject into the
# service pods as VAULT_HASHICORP_ROLE_ID / VAULT_HASHICORP_SECRET_ID.

AWS Secrets Manager

AWS Secrets Manager uses IAM-based auth. Path segments map directly to secret names. mountOS preserves the / character. Versioning uses the AWSCURRENT and AWSPREVIOUS stages.

Service env vars:

  • VAULT_PROVIDER=aws
  • VAULT_AWS_REGION
  • Credentials via standard SDK chain, or explicit VAULT_AWS_ACCESS_KEY_ID / VAULT_AWS_SECRET_ACCESS_KEY for cross-cloud use.
aws-iam-setup.sh bash download
#!/usr/bin/env bash
# mountOS AWS Secrets Manager - IAM setup
#
# Creates one IAM role per service with the least-privilege policy
# matching the permission matrix. Attach each role to the pod/instance
# running that service (via IRSA, instance profile, or workload identity).
#
# Prereqs:
#   - aws CLI configured with admin credentials
#   - REGION and ACCOUNT set below

set -euo pipefail

REGION="us-east-1"          # the Region vault's region
ACCOUNT="123456789012"      # the AWS account id
RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

# Helper: emit a policy document to stdout for a given SERVICE and access flags.
emit_policy() {
  local service="$1" volcreds_mode="$2" s3creds_mode="$3"

  local volcreds_actions='["secretsmanager:CreateSecret","secretsmanager:GetSecretValue","secretsmanager:ListSecrets"]'
  if [[ "$volcreds_mode" == "full" ]]; then
    volcreds_actions='["secretsmanager:CreateSecret","secretsmanager:GetSecretValue","secretsmanager:DeleteSecret","secretsmanager:ListSecrets"]'
  fi

  local s3creds_actions='["secretsmanager:GetSecretValue","secretsmanager:ListSecrets"]'
  if [[ "$s3creds_mode" == "full" ]]; then
    s3creds_actions='["secretsmanager:GetSecretValue","secretsmanager:CreateSecret","secretsmanager:UpdateSecret","secretsmanager:DeleteSecret","secretsmanager:ListSecrets"]'
  fi

  cat <<JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOwnSecrets",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/${service}-*"
    },
    {
      "Sid": "ReadServiceVerifiers",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue","secretsmanager:ListSecrets"],
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/service-verifiers-*"
    },
    {
      "Sid": "ReadApiMaster",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue","secretsmanager:ListSecretVersionIds"],
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/api-master-*"
    },
    {
      "Sid": "S3Creds",
      "Effect": "Allow",
      "Action": ${s3creds_actions},
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/s3creds/*"
    },
    {
      "Sid": "VolCreds",
      "Effect": "Allow",
      "Action": ${volcreds_actions},
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/volcreds/*"
    }
  ]
}
JSON
}

# appserv (Hub vault): read own + R/W on service-verifiers
cat > appserv-policy.json <<JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/appserv-*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:CreateSecret",
        "secretsmanager:UpdateSecret",
        "secretsmanager:ListSecrets"
      ],
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/service-verifiers-*"
    }
  ]
}
JSON
aws iam put-role-policy --role-name ${NAME_ROOT}-appserv --policy-name ${NAME_ROOT}-appserv --policy-document file://appserv-policy.json

# dataserv: s3creds full, volcreds create+read (no delete)
emit_policy dataserv create full > dataserv-policy.json
aws iam put-role-policy --role-name ${NAME_ROOT}-dataserv --policy-name ${NAME_ROOT}-dataserv --policy-document file://dataserv-policy.json

# gcserv: s3creds full, volcreds create+read+delete
emit_policy gcserv full full > gcserv-policy.json
aws iam put-role-policy --role-name ${NAME_ROOT}-gcserv --policy-name ${NAME_ROOT}-gcserv --policy-document file://gcserv-policy.json

# blockserv: s3creds read-only, no volcreds access
emit_policy blockserv create readonly > blockserv-policy.json
# Strip the VolCreds statement before applying (blockserv has no volcreds access)
# jq '.Statement |= map(select(.Sid != "VolCreds"))' blockserv-policy.json > blockserv-policy.fixed.json
aws iam put-role-policy --role-name ${NAME_ROOT}-blockserv --policy-name ${NAME_ROOT}-blockserv --policy-document file://blockserv-policy.json

GCP Secret Manager

GCP Secret Manager uses service-account auth. GCP secret IDs cannot contain /. The logical path uses double-underscore separators instead. For example, mountos/dataserv becomes mountos__dataserv. mountos/volcreds/<vol> becomes mountos__volcreds__<vol>.

Service env vars:

  • VAULT_PROVIDER=gcp
  • VAULT_GCP_PROJECT_ID
  • VAULT_GCP_CREDENTIALS_FILE (optional, otherwise Workload Identity or Application Default Credentials).
gcp-secret-manager-setup.sh bash download
#!/usr/bin/env bash
# mountOS GCP Secret Manager - IAM bindings
#
# Grants per-service IAM roles on each secret prefix. Bind the roles
# to workload identity / service accounts of the service pods.
#
# Prereqs:
#   - gcloud CLI authenticated with project owner / secretAdmin
#   - PROJECT set
#
# GCP secret IDs cannot contain '/', so paths use '__' separators:
#   mountos/dataserv              -> mountos__dataserv
#   mountos/volcreds/<vol>        -> mountos__volcreds__<vol>
# RESOURCE_PREFIX (optional) renames the "mountos" root to "mountos-<prefix>"
# before the same substitution, so a shared project doesn't collide.

set -euo pipefail

PROJECT="my-project"
SA_SUFFIX="@${PROJECT}.iam.gserviceaccount.com"
RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

# Every regional service reads its own bucket + replicated service-verifiers + api-master
for svc in dataserv gcserv blockserv; do
  SA="${svc}-sa${SA_SUFFIX}"
  gcloud secrets add-iam-policy-binding "${NAME_ROOT}__${svc}" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
  for prefix in service-verifiers api-master; do
    for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__${prefix}" --format="value(name)"); do
      gcloud secrets add-iam-policy-binding "$secret" \
        --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
    done
  done
done

# dataserv: admin on s3creds, create+read volcreds (no delete)
SA="dataserv-sa${SA_SUFFIX}"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__s3creds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.admin"
done
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__volcreds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretVersionAdder"
done

# gcserv: admin on s3creds AND volcreds (delete for post-deactivation cleanup)
SA="gcserv-sa${SA_SUFFIX}"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__s3creds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.admin"
done
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__volcreds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.admin"
done

# blockserv: read-only s3creds
SA="blockserv-sa${SA_SUFFIX}"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__s3creds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
done

# appserv lives in the Hub vault. Grant it admin on service-verifiers.
APPSERV_SA="appserv-sa${SA_SUFFIX}"
gcloud secrets add-iam-policy-binding "${NAME_ROOT}__appserv" \
  --member="serviceAccount:${APPSERV_SA}" --role="roles/secretmanager.secretAccessor"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__service-verifiers" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${APPSERV_SA}" --role="roles/secretmanager.admin"
done

Azure Key Vault

Azure Key Vault uses service-principal auth. Azure secret names must match [A-Za-z0-9-]+. The path uses double-dash separators instead. Azure access policies do not support per-secret ACLs. mountOS splits each path group into its own vault instead (mountos-vault-services, mountos-vault-verifiers, mountos-vault-s3creds, mountos-vault-volcreds, mountos-vault-api-master).

Service env vars:

  • VAULT_PROVIDER=azure
  • VAULT_AZURE_URL (the specific vault URL for this service path group)
  • VAULT_AZURE_TENANT_ID, VAULT_AZURE_CLIENT_ID, VAULT_AZURE_CLIENT_SECRET (optional, otherwise managed or workload identity).
azure-key-vault-setup.sh bash download
#!/usr/bin/env bash
# mountOS Azure Key Vault - access policies
#
# Azure Key Vault does not support per-secret ACLs in access policies.
# mountOS splits each path group into its own vault so that the
# service-principal access policies can be granted per vault.
#
# Azure secret names must match [A-Za-z0-9-]+, so paths use '--' as
# the separator when naming individual secrets.
#
# Prereqs:
#   - az CLI logged in with vault-admin privileges
#   - service principals created for each service; object IDs below
#   - five vaults created (one per path group)

set -euo pipefail

RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

VAULT_SERVICES="${NAME_ROOT}-vault-services"
VAULT_VERIFIERS="${NAME_ROOT}-vault-verifiers"
VAULT_API_MASTER="${NAME_ROOT}-vault-api-master"
VAULT_S3CREDS="${NAME_ROOT}-vault-s3creds"
VAULT_VOLCREDS="${NAME_ROOT}-vault-volcreds"

APPSERV_SPN="00000000-0000-0000-0000-000000000001"
DATASERV_SPN="00000000-0000-0000-0000-000000000002"
GCSERV_SPN="00000000-0000-0000-0000-000000000003"
BLOCKSERV_SPN="00000000-0000-0000-0000-000000000004"

# ---- services vault ----
# Every service reads its own secret block (names like mountos--dataserv)
for SPN in "$APPSERV_SPN" "$DATASERV_SPN" "$GCSERV_SPN" "$BLOCKSERV_SPN"; do
  az keyvault set-policy --name "$VAULT_SERVICES" \
    --object-id "$SPN" --secret-permissions get list
done

# ---- service-verifiers vault ----
# appserv writes (source of truth); others read
az keyvault set-policy --name "$VAULT_VERIFIERS" \
  --object-id "$APPSERV_SPN" --secret-permissions get set list
for SPN in "$DATASERV_SPN" "$GCSERV_SPN" "$BLOCKSERV_SPN"; do
  az keyvault set-policy --name "$VAULT_VERIFIERS" \
    --object-id "$SPN" --secret-permissions get list
done

# ---- api-master vault ----
# Write out-of-band. Regional services read only.
for SPN in "$DATASERV_SPN" "$GCSERV_SPN" "$BLOCKSERV_SPN"; do
  az keyvault set-policy --name "$VAULT_API_MASTER" \
    --object-id "$SPN" --secret-permissions get list
done

# ---- s3creds vault ----
# dataserv / gcserv: full CRUD; blockserv: read-only
az keyvault set-policy --name "$VAULT_S3CREDS" \
  --object-id "$DATASERV_SPN" --secret-permissions get set delete list
az keyvault set-policy --name "$VAULT_S3CREDS" \
  --object-id "$GCSERV_SPN" --secret-permissions get set delete list
az keyvault set-policy --name "$VAULT_S3CREDS" \
  --object-id "$BLOCKSERV_SPN" --secret-permissions get list

# ---- volcreds vault ----
# dataserv: create + read (no delete); gcserv: add delete for cleanup
az keyvault set-policy --name "$VAULT_VOLCREDS" \
  --object-id "$DATASERV_SPN" --secret-permissions get set list
az keyvault set-policy --name "$VAULT_VOLCREDS" \
  --object-id "$GCSERV_SPN" --secret-permissions get set delete list

# Azure delete uses soft-delete then purge; grant purge only if retention
# policy allows it. mountOS treats missing purge as soft-delete-only.

Permission matrix

This table lists permissions across both vaults. appserv is the hub-vault service. Every other service is regional. The table shows none for columns that do not apply to a service's vault.

ServiceOwnservice-verifiersapi-masters3credsvolcreds
appserv (common)RR/Wnonenonenone
dataservRRRCRUDCR
gcservRRRCRUDCRD
blockservRRRRnone

The mountos client has no vault entry by design. It carries no service keypair. Instead, it authenticates with an appserv-issued access-key pair. This access-key pair covers its mounts, its embedded S3 and WebHDFS gateway, and the Kubernetes CSI node driver. Its absence from the matrix is intentional, not an omission.

volcreds is write-once at the application level. dataserv and gcserv refuse to overwrite an existing volume encryption key once the volume has data. gcserv also deletes a volume's volcreds entry once three conditions are true. The volume is deactivated with vault-cleanup enabled. The grace period passes. Meta and storage cleanup finish. dataserv never deletes volcreds.

Operator responsibilities

A few operational pieces sit outside mountOS itself.

  • Provision one vault per region plus the Hub vault near the HUB. Apply the per-service policies from the matrix above.
  • Seed the expected fields for every service at install time (ED25519 key pairs, database connection values, the admin verification key in appserv).
  • Replicate service-verifiers/ from the Hub vault to every Region vault. mountOS does not ship vault-to-vault sync. Use HashiCorp performance or DR replication, AWS/GCP/Azure cross-region secret sync, or a scheduled script, according to policy.
  • Rotate ED25519 service keys on a regular schedule, typically weekly or monthly. Update the service's private key in its own path. Then update the matching public key under common/service-verifiers/<service>. Replication then pushes the new verifier out.
  • Trigger a refresh with POST /api/v1/vault/resync through the Admin SDK when a rotation lands. Rotation is zero-downtime. It needs no restart.
  • To rotate the API-key master secret, write new versions of api-master per region. gcserv picks up the new version and re-encrypts stored API keys.

Rotation cadence

WhatRotated byCadence
Service ED25519 keysoperatorWeekly or monthly, policy-driven
api-master (per region)operatorOn demand. gcserv re-encrypts stored API keys to the latest version.
Volume encryption keys (volcreds/)NeverImmutable after first write.
S3 credentials (s3creds/)operatorPer backend policy.
esc
↑↓ to navigate ↵ to open