Skip to content

Architecture

Vault and service handshake

The vault is where a mountOS deployment stores signing keys, service verifier keys, the admin verification key, per-volume encryption keys, the API-key master secret, and the credentials for the object backends behind each storage. It is required, not optional, and every service-to-service handshake reads from it.

Two vaults per deployment

mountOS uses two logical vaults: a Hub vault next to the HUB, and one Region vault per region. The two are independent, and nothing in mountOS writes across them. Replication of the shared pieces is handled out of band.

Hub vault

One per deployment, deployed alongside the HUB. Holds the HUB's own secrets and the authoritative copy of the service-verifier public keys.

  • appserv. The HUB's ED25519 signing and verification keys, the admin verification key (used to verify Admin SDK JWTs), and the admin database connection details.
  • service-verifiers. One public key per service (appserv, dataserv, gcserv, blockserv). This path is the source of truth for the whole deployment.

Region vault

One per region, deployed alongside dataserv. Holds the regional services' own secrets plus a replicated copy of the service-verifier set, so cross-service handshakes never have to call back to the HUB.

A single Region vault serves all region clusters in that region. Clustering is a volume-load pool inside the region. It does not partition the vault or the database. Every cluster in a region shares that region's one vault and one database.

  • dataserv, gcserv, blockserv. Each service's ED25519 signing and verification keys. dataserv and gcserv also keep the data database connection here.
  • service-verifiers. Replica of the Hub vault's verifier set, kept in sync by an out-of-band replication policy.
  • s3creds. Credentials for the object backends behind the region's storages (AWS, Backblaze, MinIO, and so on). Created and rotated by dataserv or gcserv.
  • volcreds. Per-volume encryption keys. Write-once at the application level, created on first use. gcserv deletes the entry after the volume is deactivated with vault-cleanup enabled and the grace period has passed.
  • api-master. The master secret that protects stored API keys. Independent per region. Rotated on a schedule, and gcserv re-encrypts stored API keys to the latest version.

The service handshake

Every service-to-service call carries a signed JWT. The receiver verifies it using the sender's public key from its local service-verifiers/ path, and rotation is zero-downtime. Regional services never have to call the HUB to verify a JWT. The replicated verifier set in their own Region vault is enough.

Keys are stored as raw base64 bytes.

Expected fields

Every vault path holds a fixed schema. Seed the service secrets at install time. The application creates s3creds and volcreds on first use. Write api-master versions as part of the API-key rotation policy. Download each template below as a starting point. Paths below show the default mountos root. An optional RESOURCE_PREFIX env var renames it to mountos-<prefix> across all four backends, so multiple deployments can share one cloud account without their secret names colliding.

appserv (Hub vault)

mountos/appserv env download
# mountOS appserv vault secrets
# Path: mountos/appserv  (Hub vault, one per deployment)
#
# Seed once at install time. Rotate the ED25519 key pair on the
# schedule defined in the rotation policy (typically weekly/monthly).
#
# All ED25519 keys are raw bytes, standard base64 (not URL-safe).
#   Signing key:      64 bytes -> 88 base64 chars
#   Verification key: 32 bytes -> 44 base64 chars
# SPKI, PEM, and PKCS8 wrappers are rejected by the verifier.

# Admin database connection
DB_DIALECT=postgresql          # mysql | postgresql
DB_URL=                        # full connection URL for the admin database
DB_PROVIDER=postgresql         # mysql | postgresql | tidb | ...
DB_PROVIDER_VERSION=           # optional, hint for capability detection (e.g. 15.2, 8.0.32)

# HUB (appserv) signing key pair
ED25519_SIGNING_KEY=           # 88 base64 chars
ED25519_VERIFICATION_KEY=      # 44 base64 chars

# Admin SDK verification key (operator ED25519 public key)
PROVIDER_VERIFICATION_KEY=     # 44 base64 chars, operator ED25519 public key

# Dashboard-user header HMAC secret. Dedicated random value (>= 32 bytes),
# distinct from PROVIDER_VERIFICATION_KEY (which is public). Signs the
# X-MountOS-Dashboard-User header; share only with the dashboard backend.
DASHBOARD_USER_HMAC_KEY=       # random secret, >= 32 bytes

dataserv (Region vault)

mountos/dataserv env download
# mountOS dataserv vault secrets
# Path: mountos/dataserv  (Region vault, one per region)
#
# Seed once at install time in every region that runs dataserv.
# Rotate the ED25519 key pair per the rotation policy.
#
# All ED25519 keys are raw bytes, standard base64.
#   Signing key:      64 bytes -> 88 base64 chars
#   Verification key: 32 bytes -> 44 base64 chars
# SPKI, PEM, and PKCS8 wrappers are rejected.

# Regional data database connection
DB_DIALECT=postgresql          # mysql | postgresql
DB_URL=                        # full connection URL for the data database
DB_PROVIDER=postgresql         # mysql | postgresql | tidb | ...
DB_PROVIDER_VERSION=           # optional, hint for capability detection

# dataserv signing key pair
ED25519_SIGNING_KEY=           # 88 base64 chars
ED25519_VERIFICATION_KEY=      # 44 base64 chars

gcserv (Region vault)

mountos/gcserv env download
# mountOS gcserv vault secrets
# Path: mountos/gcserv  (Region vault, one per region)
#
# Seed once at install time in every region that runs gcserv.
# Rotate the ED25519 key pair per the rotation policy.
#
# All ED25519 keys are raw bytes, standard base64.
#   Signing key:      64 bytes -> 88 base64 chars
#   Verification key: 32 bytes -> 44 base64 chars
# SPKI, PEM, and PKCS8 wrappers are rejected.

# Regional data database connection
DB_DIALECT=postgresql          # mysql | postgresql
DB_URL=                        # full connection URL for the data database
DB_PROVIDER=postgresql         # mysql | postgresql | tidb | ...
DB_PROVIDER_VERSION=           # optional, hint for capability detection

# gcserv signing key pair
ED25519_SIGNING_KEY=           # 88 base64 chars
ED25519_VERIFICATION_KEY=      # 44 base64 chars

blockserv (Region vault)

mountos/blockserv env download
# mountOS blockserv vault secrets
# Path: mountos/blockserv  (Region vault, one per region)
#
# Seed once at install time. Rotate per rotation policy.
# ED25519 keys are raw bytes, standard base64.

ED25519_SIGNING_KEY=           # 88 base64 chars (64 raw bytes)
ED25519_VERIFICATION_KEY=      # 44 base64 chars (32 raw bytes)

service-verifiers/<service>

mountos/service-verifiers/<service> env download
# mountOS service-verifier entry
# Path: mountos/service-verifiers/<service>
#   where <service> is one of:
#     appserv, dataserv, gcserv, blockserv
#
# The Hub vault holds the source of truth for every service's
# ED25519 public key. Replicate this path to every
# Region vault so cross-service JWT verification never has to
# call back to the HUB.
#
# Create one entry per service. Value is the matching public key
# stored in mountos/<service>/ED25519_VERIFICATION_KEY.

ED25519_VERIFICATION_KEY=      # 44 base64 chars (32 raw bytes)

api-master (Region vault)

mountos/api-master env download
# mountOS api-master entry
# Path: mountos/api-master  (Region vault, one per region)
#
# A single random string used to derive the 32-byte master key that
# encrypts API-key ciphertexts in the regional admin database.
# Independent per region.
#
# Rotation: write a new version. gcserv picks up the
# latest version via its MasterKeyAlignmentGoal and re-encrypts
# stored ciphertexts under the new master key. Services cache the
# one or two most recent versions.

key=                           # random string, at least 12 characters

s3creds/<bucket>, volcreds/<volume>

Application-managed. dataserv or gcserv writes these on first use. Grant the matrix-defined permissions on these paths, but do not seed the values.

Vault providers

Every service selects its vault with VAULT_PROVIDER plus the provider-specific credential variables. Configure one vault per region plus the Hub vault near the HUB, grant the matrix-defined permissions to each service, and replicate service-verifiers/ from the Hub vault into every region. Providers can be mixed across regions. Each vault is configured independently.

Each setup script below also accepts the same RESOURCE_PREFIX env var and applies it to every name it creates.

HashiCorp Vault

KVv2 engine mounted at mountos/. AppRole auth with automatic token renewal.

Service env vars:

  • VAULT_PROVIDER=hashicorp
  • VAULT_HASHICORP_ADDRESS
  • VAULT_HASHICORP_ROLE_ID, VAULT_HASHICORP_SECRET_ID
hashicorp-vault-setup.sh bash download
#!/usr/bin/env bash
# mountOS HashiCorp Vault setup
#
# Run the hub-vault block once on the vault near the HUB, and the
# region-vault block once per Region vault instance.
#
# Prereqs:
#   - VAULT_ADDR + VAULT_TOKEN exported (root/management token for setup)
#   - vault CLI available
#
# Policies and AppRoles defined here match the permission matrix on
# the Vault and service handshake page.

set -euo pipefail

RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

# ------------------------------------------------------------
# Hub Vault (deployed alongside the HUB, one per deployment)
# ------------------------------------------------------------

vault auth enable approle 2>/dev/null || true
vault secrets enable -path=$NAME_ROOT kv-v2 2>/dev/null || true

# appserv: read own secrets + full write on the service-verifier set
cat > appserv-policy.hcl <<EOF
path "$NAME_ROOT/data/appserv" {
  capabilities = ["read"]
}
path "$NAME_ROOT/data/service-verifiers/*" {
  capabilities = ["create", "read", "update", "list"]
}
EOF
vault policy write appserv-policy appserv-policy.hcl
vault write auth/approle/role/appserv-role token_policies="appserv-policy"

# ------------------------------------------------------------
# Region Vault (one per region, alongside the dataserv cluster)
# ------------------------------------------------------------

vault auth enable approle 2>/dev/null || true
vault secrets enable -path=$NAME_ROOT kv-v2 2>/dev/null || true

# Base policy per service: read own + replicated service-verifiers + api-master
for svc in dataserv gcserv blockserv; do
cat > ${svc}-policy.hcl <<EOF
path "$NAME_ROOT/data/${svc}" {
  capabilities = ["read"]
}
path "$NAME_ROOT/data/service-verifiers/*" {
  capabilities = ["read"]
}
path "$NAME_ROOT/data/api-master" {
  capabilities = ["read"]
}
path "$NAME_ROOT/metadata/api-master" {
  capabilities = ["read"]
}
EOF
  vault policy write ${svc}-policy ${svc}-policy.hcl
done

# s3creds: dataserv/gcserv full CRUD, blockserv read-only
cat > s3creds-full.hcl <<EOF
path "$NAME_ROOT/data/s3creds/*" {
  capabilities = ["create", "update", "read", "delete", "list"]
}
EOF
vault policy write s3creds-full s3creds-full.hcl

cat > s3creds-readonly.hcl <<EOF
path "$NAME_ROOT/data/s3creds/*" {
  capabilities = ["read", "list"]
}
EOF
vault policy write s3creds-readonly s3creds-readonly.hcl

# volcreds: dataserv create+read (write-once, no delete)
cat > volcreds-create.hcl <<EOF
path "$NAME_ROOT/data/volcreds/*" {
  capabilities = ["create", "read", "list"]
}
EOF
vault policy write volcreds-create volcreds-create.hcl

# volcreds: gcserv create+read+delete (post-deactivation cleanup)
cat > volcreds-full.hcl <<EOF
path "$NAME_ROOT/data/volcreds/*" {
  capabilities = ["create", "read", "delete", "list"]
}
EOF
vault policy write volcreds-full volcreds-full.hcl

# AppRole bindings per the matrix
vault write auth/approle/role/dataserv-role \
  token_policies="dataserv-policy,s3creds-full,volcreds-create"

vault write auth/approle/role/gcserv-role \
  token_policies="gcserv-policy,s3creds-full,volcreds-full"

vault write auth/approle/role/blockserv-role \
  token_policies="blockserv-policy,s3creds-readonly"

# Fetch role-id / secret-id per service and inject into the
# service pods as VAULT_HASHICORP_ROLE_ID / VAULT_HASHICORP_SECRET_ID.

AWS Secrets Manager

IAM-based auth. Path segments map directly to secret names with / preserved. Versioning uses the AWSCURRENT and AWSPREVIOUS stages.

Service env vars:

  • VAULT_PROVIDER=aws
  • VAULT_AWS_REGION
  • Credentials via standard SDK chain, or explicit VAULT_AWS_ACCESS_KEY_ID / VAULT_AWS_SECRET_ACCESS_KEY for cross-cloud use.
aws-iam-setup.sh bash download
#!/usr/bin/env bash
# mountOS AWS Secrets Manager - IAM setup
#
# Creates one IAM role per service with the least-privilege policy
# matching the permission matrix. Attach each role to the pod/instance
# running that service (via IRSA, instance profile, or workload identity).
#
# Prereqs:
#   - aws CLI configured with admin credentials
#   - REGION and ACCOUNT set below

set -euo pipefail

REGION="us-east-1"          # the Region vault's region
ACCOUNT="123456789012"      # the AWS account id
RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

# Helper: emit a policy document to stdout for a given SERVICE and access flags.
emit_policy() {
  local service="$1" volcreds_mode="$2" s3creds_mode="$3"

  local volcreds_actions='["secretsmanager:CreateSecret","secretsmanager:GetSecretValue","secretsmanager:ListSecrets"]'
  if [[ "$volcreds_mode" == "full" ]]; then
    volcreds_actions='["secretsmanager:CreateSecret","secretsmanager:GetSecretValue","secretsmanager:DeleteSecret","secretsmanager:ListSecrets"]'
  fi

  local s3creds_actions='["secretsmanager:GetSecretValue","secretsmanager:ListSecrets"]'
  if [[ "$s3creds_mode" == "full" ]]; then
    s3creds_actions='["secretsmanager:GetSecretValue","secretsmanager:CreateSecret","secretsmanager:UpdateSecret","secretsmanager:DeleteSecret","secretsmanager:ListSecrets"]'
  fi

  cat <<JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOwnSecrets",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/${service}-*"
    },
    {
      "Sid": "ReadServiceVerifiers",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue","secretsmanager:ListSecrets"],
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/service-verifiers-*"
    },
    {
      "Sid": "ReadApiMaster",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue","secretsmanager:ListSecretVersionIds"],
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/api-master-*"
    },
    {
      "Sid": "S3Creds",
      "Effect": "Allow",
      "Action": ${s3creds_actions},
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/s3creds/*"
    },
    {
      "Sid": "VolCreds",
      "Effect": "Allow",
      "Action": ${volcreds_actions},
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/volcreds/*"
    }
  ]
}
JSON
}

# appserv (Hub vault): read own + R/W on service-verifiers
cat > appserv-policy.json <<JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/appserv-*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:CreateSecret",
        "secretsmanager:UpdateSecret",
        "secretsmanager:ListSecrets"
      ],
      "Resource": "arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:${NAME_ROOT}/service-verifiers-*"
    }
  ]
}
JSON
aws iam put-role-policy --role-name ${NAME_ROOT}-appserv --policy-name ${NAME_ROOT}-appserv --policy-document file://appserv-policy.json

# dataserv: s3creds full, volcreds create+read (no delete)
emit_policy dataserv create full > dataserv-policy.json
aws iam put-role-policy --role-name ${NAME_ROOT}-dataserv --policy-name ${NAME_ROOT}-dataserv --policy-document file://dataserv-policy.json

# gcserv: s3creds full, volcreds create+read+delete
emit_policy gcserv full full > gcserv-policy.json
aws iam put-role-policy --role-name ${NAME_ROOT}-gcserv --policy-name ${NAME_ROOT}-gcserv --policy-document file://gcserv-policy.json

# blockserv: s3creds read-only, no volcreds access
emit_policy blockserv create readonly > blockserv-policy.json
# Strip the VolCreds statement before applying (blockserv has no volcreds access)
# jq '.Statement |= map(select(.Sid != "VolCreds"))' blockserv-policy.json > blockserv-policy.fixed.json
aws iam put-role-policy --role-name ${NAME_ROOT}-blockserv --policy-name ${NAME_ROOT}-blockserv --policy-document file://blockserv-policy.json

GCP Secret Manager

Service-account auth. GCP secret IDs cannot contain /, so the logical path uses double-underscore separators (mountos/dataserv becomes mountos__dataserv, mountos/volcreds/<vol> becomes mountos__volcreds__<vol>).

Service env vars:

  • VAULT_PROVIDER=gcp
  • VAULT_GCP_PROJECT_ID
  • VAULT_GCP_CREDENTIALS_FILE (optional, otherwise Workload Identity or Application Default Credentials).
gcp-secret-manager-setup.sh bash download
#!/usr/bin/env bash
# mountOS GCP Secret Manager - IAM bindings
#
# Grants per-service IAM roles on each secret prefix. Bind the roles
# to workload identity / service accounts of the service pods.
#
# Prereqs:
#   - gcloud CLI authenticated with project owner / secretAdmin
#   - PROJECT set
#
# GCP secret IDs cannot contain '/', so paths use '__' separators:
#   mountos/dataserv              -> mountos__dataserv
#   mountos/volcreds/<vol>        -> mountos__volcreds__<vol>
# RESOURCE_PREFIX (optional) renames the "mountos" root to "mountos-<prefix>"
# before the same substitution, so a shared project doesn't collide.

set -euo pipefail

PROJECT="my-project"
SA_SUFFIX="@${PROJECT}.iam.gserviceaccount.com"
RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

# Every regional service reads its own bucket + replicated service-verifiers + api-master
for svc in dataserv gcserv blockserv; do
  SA="${svc}-sa${SA_SUFFIX}"
  gcloud secrets add-iam-policy-binding "${NAME_ROOT}__${svc}" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
  for prefix in service-verifiers api-master; do
    for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__${prefix}" --format="value(name)"); do
      gcloud secrets add-iam-policy-binding "$secret" \
        --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
    done
  done
done

# dataserv: admin on s3creds, create+read volcreds (no delete)
SA="dataserv-sa${SA_SUFFIX}"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__s3creds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.admin"
done
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__volcreds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretVersionAdder"
done

# gcserv: admin on s3creds AND volcreds (delete for post-deactivation cleanup)
SA="gcserv-sa${SA_SUFFIX}"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__s3creds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.admin"
done
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__volcreds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.admin"
done

# blockserv: read-only s3creds
SA="blockserv-sa${SA_SUFFIX}"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__s3creds" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${SA}" --role="roles/secretmanager.secretAccessor"
done

# appserv lives in the Hub vault. Grant it admin on service-verifiers.
APPSERV_SA="appserv-sa${SA_SUFFIX}"
gcloud secrets add-iam-policy-binding "${NAME_ROOT}__appserv" \
  --member="serviceAccount:${APPSERV_SA}" --role="roles/secretmanager.secretAccessor"
for secret in $(gcloud secrets list --filter="name:${NAME_ROOT}__service-verifiers" --format="value(name)"); do
  gcloud secrets add-iam-policy-binding "$secret" \
    --member="serviceAccount:${APPSERV_SA}" --role="roles/secretmanager.admin"
done

Azure Key Vault

Service-principal auth. Azure secret names must match [A-Za-z0-9-]+, so the path uses double-dash separators. Per-secret ACLs are not native in access policies, so mountOS splits each path group into its own vault (mountos-vault-services, mountos-vault-verifiers, mountos-vault-s3creds, mountos-vault-volcreds, mountos-vault-api-master).

Service env vars:

  • VAULT_PROVIDER=azure
  • VAULT_AZURE_URL (the specific vault URL for this service path group)
  • VAULT_AZURE_TENANT_ID, VAULT_AZURE_CLIENT_ID, VAULT_AZURE_CLIENT_SECRET (optional, otherwise managed or workload identity).
azure-key-vault-setup.sh bash download
#!/usr/bin/env bash
# mountOS Azure Key Vault - access policies
#
# Azure Key Vault does not support per-secret ACLs in access policies.
# mountOS splits each path group into its own vault so that the
# service-principal access policies can be granted per vault.
#
# Azure secret names must match [A-Za-z0-9-]+, so paths use '--' as
# the separator when naming individual secrets.
#
# Prereqs:
#   - az CLI logged in with vault-admin privileges
#   - service principals created for each service; object IDs below
#   - five vaults created (one per path group)

set -euo pipefail

RESOURCE_PREFIX="${RESOURCE_PREFIX:-}"
NAME_ROOT="mountos${RESOURCE_PREFIX:+-$RESOURCE_PREFIX}"

VAULT_SERVICES="${NAME_ROOT}-vault-services"
VAULT_VERIFIERS="${NAME_ROOT}-vault-verifiers"
VAULT_API_MASTER="${NAME_ROOT}-vault-api-master"
VAULT_S3CREDS="${NAME_ROOT}-vault-s3creds"
VAULT_VOLCREDS="${NAME_ROOT}-vault-volcreds"

APPSERV_SPN="00000000-0000-0000-0000-000000000001"
DATASERV_SPN="00000000-0000-0000-0000-000000000002"
GCSERV_SPN="00000000-0000-0000-0000-000000000003"
BLOCKSERV_SPN="00000000-0000-0000-0000-000000000004"

# ---- services vault ----
# Every service reads its own secret block (names like mountos--dataserv)
for SPN in "$APPSERV_SPN" "$DATASERV_SPN" "$GCSERV_SPN" "$BLOCKSERV_SPN"; do
  az keyvault set-policy --name "$VAULT_SERVICES" \
    --object-id "$SPN" --secret-permissions get list
done

# ---- service-verifiers vault ----
# appserv writes (source of truth); others read
az keyvault set-policy --name "$VAULT_VERIFIERS" \
  --object-id "$APPSERV_SPN" --secret-permissions get set list
for SPN in "$DATASERV_SPN" "$GCSERV_SPN" "$BLOCKSERV_SPN"; do
  az keyvault set-policy --name "$VAULT_VERIFIERS" \
    --object-id "$SPN" --secret-permissions get list
done

# ---- api-master vault ----
# Write out-of-band. Regional services read only.
for SPN in "$DATASERV_SPN" "$GCSERV_SPN" "$BLOCKSERV_SPN"; do
  az keyvault set-policy --name "$VAULT_API_MASTER" \
    --object-id "$SPN" --secret-permissions get list
done

# ---- s3creds vault ----
# dataserv / gcserv: full CRUD; blockserv: read-only
az keyvault set-policy --name "$VAULT_S3CREDS" \
  --object-id "$DATASERV_SPN" --secret-permissions get set delete list
az keyvault set-policy --name "$VAULT_S3CREDS" \
  --object-id "$GCSERV_SPN" --secret-permissions get set delete list
az keyvault set-policy --name "$VAULT_S3CREDS" \
  --object-id "$BLOCKSERV_SPN" --secret-permissions get list

# ---- volcreds vault ----
# dataserv: create + read (no delete); gcserv: add delete for cleanup
az keyvault set-policy --name "$VAULT_VOLCREDS" \
  --object-id "$DATASERV_SPN" --secret-permissions get set list
az keyvault set-policy --name "$VAULT_VOLCREDS" \
  --object-id "$GCSERV_SPN" --secret-permissions get set delete list

# Azure delete uses soft-delete then purge; grant purge only if retention
# policy allows it. mountOS treats missing purge as soft-delete-only.

Permission matrix

Permissions across both vaults. appserv is the hub-vault service. Everyone else is regional. Columns that do not apply to a service's vault are listed as none.

ServiceOwnservice-verifiersapi-masters3credsvolcreds
appserv (common)RR/Wnonenonenone
dataservRRRCRUDCR
gcservRRRCRUDCRD
blockservRRRRnone

The mountos client has no vault entry by design. It carries no service keypair and authenticates with an appserv-issued access-key pair instead, and that covers its mounts, its embedded S3 and WebHDFS gateway, and the Kubernetes CSI node driver. Its absence from the matrix is intentional, not an omission.

volcreds is write-once at the application level. dataserv and gcserv refuse to overwrite an existing volume encryption key once the volume has data. gcserv additionally deletes a volume's volcreds entry after the volume has been deactivated with vault-cleanup enabled, its grace period has passed, and meta and storage cleanup have already completed. dataserv never deletes volcreds.

Operator responsibilities

A few operational pieces sit outside mountOS itself.

  • Provision one vault per region plus the Hub vault near the HUB. Apply the per-service policies from the matrix above.
  • Seed the expected fields for every service at install time (ED25519 key pairs, database connection values, the admin verification key in appserv).
  • Replicate service-verifiers/ from the Hub vault to every Region vault. mountOS does not ship vault-to-vault sync. Use HashiCorp performance or DR replication, AWS/GCP/Azure cross-region secret sync, or a scheduled script, according to policy.
  • Rotate ED25519 service keys on a regular schedule, typically weekly or monthly. Update the service's private key in its own path, then update the matching public key under common/service-verifiers/<service>. Replication then pushes the new verifier out.
  • Trigger a refresh with POST /api/v1/vault/resync through the Admin SDK when a rotation lands. Rotation is zero-downtime and no restart is required.
  • Write new versions of api-master per region when rotating the API-key master secret. gcserv picks up the new version and re-encrypts stored API keys.

Rotation cadence

WhatRotated byCadence
Service ED25519 keysoperatorWeekly or monthly, policy-driven
api-master (per region)operatorOn demand. gcserv re-encrypts stored API keys to the latest version.
Volume encryption keys (volcreds/)NeverImmutable after first write.
S3 credentials (s3creds/)operatorPer backend policy.
esc
↑↓ to navigate ↵ to open