# mountOS Full AI-Agent Documentation

> Self-hosted, POSIX-compatible distributed storage with native macOS, Linux, and Windows mounts plus S3 and HDFS, and built-in forks, time travel, and one-minute versioning.

Generated bundle. Do not edit; edit topics/ and rebuild.


<!-- topic: overview -->

# Overview

mountOS is self-hosted cloud storage infrastructure. It gives a team native filesystem access across macOS, Linux, and Windows over the same data, plus a standard S3-compatible API for tools that do not mount a filesystem. Both the metadata store and the file data live on infrastructure the operator runs, not on a vendor's cloud.

## What it is

- A distributed filesystem you self-host, not a hosted SaaS.
- Fully POSIX-compatible, so existing applications and tools run unchanged. Compatibility is verified with the industry filesystem suites, each covering a different surface: `pjdfstest` for POSIX compliance (8,789 tests), the Linux Test Project (`LTP`) for filesystem and system-call behavior, `xfstests` for filesystem regressions, `fsx` for I/O consistency under randomized operations, and `IOR` with `mdtest` for I/O throughput and metadata load.
- Metadata lives in a MySQL or PostgreSQL wire-compatible database (operator's choice of engine).
- File content lives in S3-compatible or Azure object storage (AWS S3, Google Cloud Storage, Backblaze B2, MinIO, Azure Blob Storage, or any S3-compatible on-prem tier), or on local block devices for self-managed clusters.
- One wire protocol drives every filesystem client, so a mount behaves the same on each platform.

## Value proposition

- Own the data and the infrastructure. A volume lives in exactly one region and its content does not leave that region at serving time.
- Point each plane (metadata, data) at infrastructure the operations team already trusts.
- One interface per native filesystem surface, so users keep the tools they already know (Finder, POSIX paths, Windows Explorer), while S3-native tooling reaches the same volumes without a mount.

## Headline capabilities

These are built in, with no extra subsystem to operate.

### Forks

A fork is a layered view of an entire volume. Creating one does not copy file data or duplicate the existing tree; the fork starts from the same underlying history and diverges only as new writes land.

- Branch a whole volume git-style in one call, then mount the fork on any client.
- A fork can itself be forked. Nesting is bounded up to eight levels deep so chains stay predictable.
- Writes on a fork stay on that fork. Shared blocks diverge only when they change.
- Forks do not merge back. A fork is a fresh line of history; keep the one you want.
- Discard a fork in one call. Nothing was copied, so nothing has to be collected on the hot path.
- Use cases include build isolation, experimentation, dataset branches, and CI workspaces.

Temporary forks exist only for the duration of a mount session. Their writes are stored on the client's local disk and bypass the object backend, so no file data reaches S3 or any other configured backend. Metadata still flows through the Meta server. When the volume is unmounted, the temporary fork's metadata is cleaned up shortly after.

### Time travel

Open a volume as it looked at an earlier point in time by picking a timestamp.

- Browse the volume as of a minute ago, an hour ago, or last Tuesday.
- Read any file from that moment, copy it out, or diff it against now.
- Granularity matches the version history, down to one minute.
- No periodic snapshots, side copies, or separate subsystem to operate.

### Version history (one-minute versioning)

Every file keeps a rolling history at one-minute granularity. If a file is saved more than once within the same minute, the intermediate versions are collected automatically so the history stays useful rather than noisy.

- List versions for any file, open or diff any version.
- Restore an older version in place, or copy it out.

### Recently deleted

Deleted files and folders stay available in a recently-deleted view for a configurable window, restorable without pulling a backup.

- Restores are instant and metadata-only.
- Works the same for a single file, a folder, or a subtree.
- Entries age out after the window closes.
- The window is the volume's retention period (default 30 days, configurable from 0 to 366 days).

### Retention

Retention is one knob per volume, set in the admin console (the `retentionPeriod` field on a volume). It controls how far back time travel reaches, how long deleted files stay restorable, and when old versions and tombstones age out of the database and the object backend. The default is 30 days, and it can be set anywhere from 0 to 366 days.

- One setting covers version history, recently-deleted, and final cleanup.
- An active fork extends the floor, so retention reaches back to the oldest snapshot any fork still points at and data a fork can still see is never pruned.
- Temporary forks do not extend retention; they pin at the current moment and vanish on unmount.
- Volume deactivation has its own grace period (the `gracePeriod` field), independent of day-to-day retention. It defaults to 14 days and can be set from 0 to 91 days; after it elapses, gcserv clears the deactivated volume's secrets and reaps its remaining data.
- A separate fork grace period (`forkGracePeriod`) governs how long a discarded fork's data lingers before cleanup. It defaults to 1 day and can be set from 0 to 30 days.

### Change events

Every general volume can keep an ordered feed of its changes (create, delete,
modify, rename), per fork, with full paths and monotonic sequence numbers. The
`mountos event` command reads it as a stream on stdout or serves it as a local REST API.

- Off by default; one per-volume knob (`eventLogRetentionPeriod`, 0-30 days) gates capture
  and sets the readable window.
- Tail live, replay a time window, or filter to a subtree, server side.
- Built for indexers, sync tools, and AI agents that follow a volume without walking the tree.
- General volumes only; near real time (capture flushes about every 3 seconds).

### AI MCP connector

Each mountOS client ships a local, read-only MCP (Model Context Protocol) connector. An AI agent can attach to a running mount and inspect it without touching file contents: identity, live stats, throughput, recent deletions, recovered (lost-and-found) files, configuration, diagnostics, and the CLI commands that operate it. The connector never reads file data and never mutates state, so it is safe to run against production mounts.

## OS-layer and protocol support matrix

Anything that speaks S3 or WebHDFS (CLI tools, SDKs, build systems, CI jobs) can also read and write the same data through the client's embedded gateway with no filesystem mount.

Every surface below ships.

| Platform | Surface | Client / mechanism |
|----------|---------|--------------------|
| macOS | FSKit | mountOSX (native filesystem extension, macOS 15.4 and up) |
| macOS | macFUSE | fallback for older macOS |
| macOS | NFS 4.1 / SMB 3 | direct client |
| Linux | FUSE | mountos (libfuse3), the main Linux surface |
| Linux | Kubernetes CSI | `mountos kubernetes` node plugin (spawns one mountos mount per volume) |
| Linux | NFS (NFSv4) | direct client |
| Windows | mountosio | mountos-win-driver (native kernel filesystem driver) |
| Protocol | S3 REST | any S3 client, AWS SigV4, through `mountos gateway --gateway s3` |
| Protocol | WebHDFS | stock Hadoop clients, through `mountos gateway --gateway hdfs` |
| SDK | Admin SDK (Go) | account/volume/key management, ED25519 JWT |
| SDK | Admin SDK (TS) | account/volume/key management, ED25519 JWT |
| SDK | Admin SDK (Rust) | account/volume/key management, ED25519 JWT |

## How the planes split

mountOS divides cleanly into a metadata plane and a data plane.

- Meta engine (Meta server). Holds the file tree, versions, forks, and access rules. A clustered in-memory arena cache serves live reads, backed by the database so the working set stays hot and the database takes far fewer hits. The cluster synchronizes volume ownership through raft. MySQL or PostgreSQL wire-compatible.
- Storage (Block Storage Server, blockserv, optional). The block-storage byte plane and an alternative to a cloud object backend. A block storage is up to three active-active block volumes that share a storage id and form a peer mesh across distinct clusters (no primary, no failover), each a blockserv node that caches object-storage parts on a raw block device. Clients reach it over the custom, encrypted block protocol, not S3. It can proxy an external object store or serve bytes from local block devices. File content lands in S3, MinIO, GCS, Backblaze B2, any S3-compatible tier, or local block devices.
- Embedded gateway (part of the `mountos` client, optional). `mountos gateway --gateway s3,hdfs` exposes one volume over the S3 REST contract (buckets, listings, multipart, lifecycle) and over WebHDFS, on a local endpoint, with or without a filesystem mount. Nothing extra is deployed server-side; the gateway runs wherever the client runs.
- gcserv (housekeeping). Collects orphaned blocks, prunes old forks past the retention floor, and clears volume secrets after a deactivated volume's grace period (default 14 days). Runs as its own pool per cluster, scaled independently of the serving path.

## Topology at a glance

- One global HUB (scaled to as many instances as load needs) answers discovery ("which region owns this volume?"), serves the admin APIs behind the Admin SDKs, and holds global account, volume, and access-key records.
- Any number of independent regions serve the actual data. A region runs the Meta server cluster, optional block storages (each up to three active-active block volumes spanning distinct clusters), and a region database, and points at one or more storages, records naming an S3-compatible or Azure store (endpoint, credentials, configuration), ideally in the same locality.
- Clients hit the HUB once to discover their region, then talk to that region for the life of the session.

### Auth and wire protocols

| Surface | Protocol | Auth |
|---------|----------|------|
| Filesystem clients <-> Meta server | custom encrypted protocol over TCP | S3-style access key / secret pair |
| S3 clients <-> embedded gateway | HTTP or HTTPS, S3 REST | AWS SigV4 access key |
| Hadoop clients <-> embedded gateway | HTTP or HTTPS, WebHDFS | mountOS SigV4 access key (service `hdfs`) |
| Clients <-> HUB (discovery) | HTTPS, JSON | None, cacheable |
| Admin SDKs <-> HUB | HTTPS, JSON | ED25519-signed JWT |

Filesystem clients complete an encrypted handshake for a mutually-authenticated session, then present an S3-style access key and secret that pins account, storage, quota, and encryption per volume for the session. Service-to-service requests use Ed25519-signed JWTs verified against a trusted service-verifier set. The deployment requires a vault for secrets; supported vault backends include HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault.

## Guarantees and non-guarantees

What mountOS promises, stated at the contract level (the site page is /docs/guarantees):

- POSIX semantics on every POSIX mount, verified with pjdfstest (8,789 tests), LTP, xfstests, and fsx, and exercised under load with IOR and mdtest.
- A volume lives in exactly one region. No cross-region replication.
- Metadata is durable in the region database. The dataserv cluster uses Raft only to synchronize volume ownership (3 nodes tolerate 1 loss, 5 tolerate 2). File content inherits the durability of the store the volume points at.
- A HUB outage does not interrupt active sessions; first-time discovery fails until the HUB returns, previously mounted clients continue on cached topology.
- Registration self-heals: a misconfigured service starts, refuses to register, and retries every minute; a mismatched or deactivated region and cluster pair never registers, and the HUB raises a critical topology alert when it rejects a registration that reaches it.
- Version history at one-minute granularity; within the retention window restores are guaranteed, and deleted-entry restores are instant and metadata-only; data an active fork can still see is never pruned.
- Fork isolation: writes on a fork stay on the fork, forks never merge back, temporary forks never write file data to the backend.
- Changes made through one client are visible to other mounts; metadata is served from the owning cluster and file content is immutable, so mounts share one view.
- Not supported: cross-region serving (mountOS is region-based), sub-minute version granularity.

## Licensing and plans

mountOS is licensed per logical unit: one complete, independently operated deployment and the storage it manages. Paid licenses are issued as signed tokens and managed through the HUB, which holds the active set and serves it to every service. A deployment with no license runs free on a built-in default tier, self-hosted at no charge for a single deployment, up to a default capacity (currently 10 TiB) that needs no token to install. A Business License, arranged through an Order Form, raises capacity and adds support. The default capacity is always included on top of any license, so a license adds its capacity rather than replacing the default. Each additional logical unit requires its own license. Every capability is available in every plan. Plans differ on capacity and support, not features. The full agreement is published at https://mountos.io/terms.

## Is mountOS a fit?

Good fit when a team wants:

- Self-hosted storage with native filesystem access on macOS, Linux, and Windows plus an S3 API over the same data.
- Built-in version history, time travel, and recently-deleted restore without scheduling snapshots.
- Git-style volume forks for CI, experimentation, or dataset branching.
- Control over which database engine and which object/block storage backend hold metadata and data.
- Kubernetes integration via a CSI node plugin.

Less of a fit when a fully managed, vendor-operated storage service is preferred, or when cross-region replication of a single volume's serving path is required (a volume is single-region at serving time).


<!-- topic: architecture -->

# Architecture

mountOS is a distributed filesystem and object platform. A small set of server services hold metadata and run background maintenance, while clients mount volumes through native OS filesystem interfaces or reach them through S3 and HDFS gateways. All inter-service traffic runs over a custom encrypted, mutually authenticated RPC transport.

## High-level shape

A mountOS deployment separates three planes.

- Control and metadata plane. The server services hold object metadata, authentication, accounts, volumes, and topology. Data content (file bytes) is stored as segments/blobs, and the metadata services track structure, versioning, and access.
- Client plane. Endpoints mount a volume through an OS-native filesystem backend (FUSE, FSKit, NFS, SMB, or the Windows kernel driver) and speak the binary metadata protocol to the metadata service.
- Gateway plane. The client can expose the volume it is bound to over S3 and WebHDFS, so existing tools reach the same data without a mount. The gateway is a feature of the client process, not a separately deployed service.

## Server services

Each service is a separate binary built from a single codebase. Roles:

| Service | Role |
| --- | --- |
| `appserv` | Auth and application services (accounts, teams, volume and storage management). Also the central service discovery hub. |
| `dataserv` | Metadata backbone. Owns object metadata, directory trees, forks, versions, ACLs/xattrs, and segment/blob bookkeeping. Serves the binary metadata protocol to clients and gateways. |
| `gcserv` | Data cleanup and repair (periodic removal of orphaned data and temporary files, maintenance work goals, and lake compaction). |
| `blockserv` | Optional block-storage byte plane, an alternative to a cloud object backend. A block storage is up to three active-active block volumes (each a `blockserv` node with `BLOCK_VOLUME_ID`, bound to its `REGION_CLUSTER_ID`) that form a peer mesh across distinct clusters, caching object-storage parts on a raw block device (preferably XFS-formatted) with object storage as the durable source of truth. Clients reach it over the custom, encrypted block protocol, not S3. It proxies an external object store, or is storage on local block devices. |

`mountos` is the client binary rather than a server service. It implements the OS filesystem backends, the embedded S3/WebHDFS gateway, and the Kubernetes CSI node driver (`mountos kubernetes`, Linux only, exposing the Identity, Controller, and Node CSI gRPC services and launching one mount subprocess per staged volume).

The embedded gateway serves general volumes. Volumes typed as iceberg/lake are not served over S3 or WebHDFS. Lake (Apache Iceberg) volumes are reached through `mountos` and a lake REST catalog instead, which the client starts automatically for that volume type.

## Regions and clusters

mountOS partitions deployments hierarchically.

- A region is a deployment scope with its own database and secret store. Services register and operate within a region.
- A cluster partitions workload inside a region (selected per node via the `REGION_CLUSTER_ID` setting). Services that are cluster-scoped (such as `dataserv`, `gcserv`'s per-volume goals, and the optional `blockserv`) only handle the shards assigned to their cluster, while region-wide singleton work stays unpartitioned. This lets one region host multiple independent workload partitions sharing the same backing database and vault.

Volumes can be moved between clusters (the `volumes.move_cluster` flow), and clients prime a volume-to-owner topology cache so they route requests to the correct owning node. A cluster move is not an instant cutover. The source cluster keeps serving the volume and stamps a cluster hint on responses until a handover deadline (a `HandoverUntil` timestamp in unix seconds), after which the destination cluster takes ownership. A background handover sweeper clears the transient handover state once the window expires.

## Ownership and routing (dataserv)

`dataserv` uses Raft for ownership of metadata. Ownership is keyed by `(volume, fork)`, so one owner node serves all inodes for a given fork and writes are routed to that owner. Metadata is held in an in-memory arena cache backed by the database. When a request reaches a non-owner, the node forwards it and returns an ownership hint so the client can route directly next time. The metadata protocol envelope carries routing fields (volume id, fork id, parent-dir partition hint) and the server stamps ownership hints onto responses.

## Client protocol backends

`mountos` is the single client binary, and the active backend depends on platform and flags. Each backend presents a mounted volume to the OS and translates VFS operations into the metadata protocol.

| Backend | Platform | Selection |
| --- | --- | --- |
| `fuse` / `fuse+iouring` | Linux | default Linux mount |
| macFUSE | macOS | `--macfuse` |
| FSKit | macOS | `--fskit` / `-k` (FSKit filesystem extension, macOS 15.4+) |
| NFS | macOS | `--nfs` |
| SMB | macOS | `--smb` |
| `mountosio` | Windows | default (and only) Windows backend, a native kernel filesystem driver |

Backends are mutually exclusive, and only one is selected per mount. On macOS the backend can be auto-selected when no explicit flag is given.

Mount-option convention: standard mount options pass through `-o` (see `man 8 mount`), and custom/application options (debug, umask, permission tuning) use a separate mechanism such as URL query parameters. The two are never mixed.

### Embedded gateway

`mountos` can additionally run an embedded gateway via the `--gateway` flag (comma-separated protocols, for example `s3`, `hdfs`, or `s3,hdfs`). This exposes the same volume over the S3/HDFS APIs directly from the client process. By default the embedded gateway binds loopback (`127.0.0.1`), and binding non-loopback requires a TLS cert and key. Iceberg/lake volumes are served through `mountos` plus a lake REST catalog that implements the Iceberg REST spec. In that mode the file body bytes never traverse the catalog RPC, only metadata and segment references.

## Request flow

A typical metadata request:

1. A POSIX operation (open, read, lookup, write) arrives at the OS backend in `mountos`.
2. `mountos` resolves the target volume and fork, then routes to the owning `dataserv` node using its cached topology (falling back to forwarding plus hints if the cache is stale).
3. The request is sent over an encrypted connection to the owning node.
4. `dataserv` serves it from the arena cache (backed by the database), applying Raft-routed writes on the owner.
5. The response carries the result plus optional routing/throttle hints.

Gateway requests (S3/HDFS) follow the same backend path. The gateway authenticates the caller, resolves the path/volume, and issues the same metadata-protocol calls to `dataserv`.

## Transport

Inter-service communication uses a custom encrypted protocol over TCP that mutually authenticates both peers before any application data flows. Peers are first-party mountOS service binaries. Requests and responses carry a method name (for example `discovery.register`, `volumes.create`, `tree.list`), a correlation id, a per-request timeout, and an error field.

- Bidirectional. A single persistent connection supports outbound calls and inbound (reverse) calls dispatched to registered handlers, so a server can call back into a connected client (for example cache-invalidation events).
- Resilience. Clients keep a persistent connection and retry once on transport failure by reconnecting. Application-level errors are returned without reconnect.

Method names group by domain: discovery (`discovery.register`, `discovery.heartbeat`, `discovery.deregister`, `discovery.resolve`), region/cluster resolution (`region.resolve`, `region_cluster.resolve`), storages, volumes/forks, accounts, quota, raft (`raft.*`), alerts, audit logs, admin tree browse (`tree.*`), and the lake/Iceberg catalog (`lake.*`, for example `lake.namespaces.list`, `lake.tables.load`).

Two message families coexist on the wire: the higher-level service control RPCs and the metadata protocol, a rich typed filesystem protocol with many request/response message types covering directory listing, entry create/lookup, blobs and segments, xattr/ACL, locks, forks, snapshots, multipart, POSIX compound mutations, path resolution, and the lake catalog.

The optional `blockserv` byte plane is the exception. Bulk byte I/O uses a separate custom, encrypted block protocol tuned for streaming rather than request/response, so it does not share the internal RPC envelope.

## Service discovery

`appserv` is the central discovery hub. Server nodes self-register and heartbeat (the `discovery.register` / `discovery.heartbeat` / `discovery.deregister` / `discovery.resolve` methods, persisted in a service-nodes registry). Nodes resolve their advertised public address either from an explicit advertise address or via cloud instance metadata (AWS, GCP, Azure, DigitalOcean, Hetzner, OCI auto-detected). Client-facing discovery of the metadata endpoint is a separate, lighter path.

## Authentication boundaries

- Clients and gateways present access keys, and the S3 and HDFS gateways verify AWS SigV4-compatible signed requests (HDFS uses mountOS SigV4 with service name `hdfs`). The access key encodes routing identity (a parseable key carrying user and volume-shard identifiers).
- The gateway is the trust boundary for external callers. Inter-service peers are first-party binaries authenticated by the signed handshake, so server handlers focus on resource limits and internal preconditions rather than defensive input validation.


<!-- topic: components -->

# Components

This page catalogs the deployable server services, the per-OS mount clients, and the SDKs and admin tooling a third party integrates with. It does not cover internal package layout.

## Server services

mountOS servers are a Linux-oriented set of four services built from one monorepo (`mountos-servers`). Each service is selected at build time and runs as its own process. Services talk to clients over a custom encrypted protocol over TCP, and to each other over an internal RPC.

| Service | Role | When it is involved |
| --- | --- | --- |
| `appserv` | Auth and management plane, the global HUB. Handles authentication, account/user/volume/team management, issues the S3 access key pairs that gateways use, and serves discovery (clients talk to it once to find their region, then talk to the region directly). | Always. Provisioning users, volumes, teams, access keys; admin operations; discovery. |
| `dataserv` | Metadata backbone (the "Meta server"). Holds the directory tree, forks, versions, and partition ownership, with a hot working set in an in-memory arena cache written through to the region database. Raft synchronizes volume ownership across the cluster. Mount clients connect to it over the binary TCP protocol. | Always. Every file/metadata operation from a mount client. |
| `blockserv` | Block Storage Server (optional). The block-storage byte plane and an alternative to a cloud object backend. A block storage is 1 (Single) or 2 to 3 active-active block volumes (HA) sharing a `storage_id`, forming a peer mesh across distinct clusters (no primary, no failover); members find each other by `storage_id` at runtime. Each block volume is one `blockserv` node (`BLOCK_VOLUME_ID`, bound to its `REGION_CLUSTER_ID`) that caches object-storage parts on a raw block device (preferably XFS-formatted). Clients reach it over the custom, encrypted block protocol, not S3; it can proxy an external object store or be storage on local block devices. | Optional byte I/O; volumes on a cloud object backend do not need it. |
| `gcserv` | Garbage collection, data cleanup, and repair. Runs periodic cleanup of orphaned data, temporary files, retention/compaction (including Iceberg compaction), and database maintenance. Runs as its own pool per cluster and scales independently. | Background maintenance. Not on the request hot path. |

S3, WebHDFS, and Kubernetes are not server services. They are modes of the `mountos` client binary (`mountos gateway` and `mountos kubernetes`), covered below.

### Default listener ports

Ports are configured per deployment via environment variables, so these are defaults, not fixed values.

| Service | Surface | Default port | Configured by |
| --- | --- | --- | --- |
| `dataserv` | Binary TCP (mount-client protocol) | 6464 | `APP_PORT` |
| `blockserv` | Block protocol byte I/O (peers on `BLOCK_PORT`+1) | 9100 | `BLOCK_PORT` |

The region services that bind a TCP/HTTP listener also expose an internal RPC port for service-to-service RPC, configured by `RPC_PORT` (derived from `APP_PORT` when unset; for example dataserv defaults its internal RPC port to `APP_PORT + 2`). The appserv management-plane HTTP port is set by `PORT` per deployment.

Client-side listeners (the embedded gateway and the CSI health endpoint) are covered with the client below.

## Mount clients and backends per OS

The filesystem clients let an OS mount a mountOS volume as a local filesystem. macOS ships several backend options; Linux uses FUSE; Windows uses a native kernel driver.

### macOS

macOS support is delivered as a desktop application bundle (`mountos-osx`, app target `mountOSX`) plus several mount backends. The app embeds the FSKit extension and bridges to the other backends. The FSKit and FUSE-style backends target macOS 15.4 or later (the FUSE build sets a 15.4 deployment target).

| Component / repo | What it is | When you need it |
| --- | --- | --- |
| `mountos-osx` (`mountOSX` app) | The macOS desktop app and tray UI that manages mounts and bundles the backends. | Primary macOS install. |
| `mountos-fskit` | FSKit-based filesystem extension, the modern Apple file-system framework backend. | macOS 15.4+ FSKit mounts (default modern path). |
| `mountos-fuse` | Clean-room FUSE 3.x compatible library for macOS, with XPC IPC to the FSKit extension. Not libfuse (its README describes it as a proprietary implementation based on public API specifications). macOS 15.4+. | macFUSE-style FUSE mounts on macOS. |
| `mountos-nfs` (`mountos-nfs-osx`) | NFS server backend for macOS. | Mounting a volume over NFS on macOS. |
| `mountos-smb-osx` | SMB server backend for macOS. | Mounting a volume over SMB on macOS. |

### Linux

| Component | What it is | When you need it |
| --- | --- | --- |
| `mountos` | The mountOS FUSE filesystem client binary. Mounts remote workspaces as local filesystems and also hosts `mountos gateway`. Installed at a path such as `/usr/local/bin/mountos`. | Primary Linux mount client; also the binary the CSI driver invokes. |
| `mountos-linux` | Linux client packaging/integration for mountOS. | Linux client distribution. |

The `mountos` binary is a Cobra CLI. Its subcommands include `mount`, `unmount`, `gateway`, `info`, `list`, `control`, `stats`, `locks`, `volume`, `dashboard` (TUI by default, `--gui` for browser), `snapshot` (read-only point-in-time mount), `deleted` (mount showing only deleted files), `fork` (list/create named forks), `event`, `repair`, `check`, `bench`, `log`, `stream`, `systemd` (prints systemd unit templates), `version`, and `man`. It also supports being invoked via `mount.mountos` for standard `mount -o` integration. The published client package is `mountos-cli` on every platform.

### Windows

| Component | What it is | When you need it |
| --- | --- | --- |
| `mountos-win-driver` (`MountOsIo`, mountosio) | Windows WDM kernel-mode filesystem driver written in Zig. Targets the Windows 11 / Windows Server 2025 kernel family, WDK 10.0.26100+. The Windows backend. | Native kernel filesystem mounts on Windows. |

## Embedded S3 and WebHDFS gateway

S3 and WebHDFS are protocol surfaces of the `mountos` client, bound to the one volume the client is authenticated for. There is no multi-tenant gateway fleet to deploy.

```
mountos gateway --gateway s3        -a <access-key-id> -s
mountos gateway --gateway hdfs      -a <access-key-id> -s
mountos gateway --gateway s3,hdfs   -a <access-key-id> -s --gateway-port 0
```

- `mountos gateway` runs the gateway with no filesystem mount. Passing `--gateway s3,hdfs` to `mountos mount` instead runs both a mount and the gateway from the same process.
- `--gateway-port` accepts `0` (pick a free port per protocol, the default), a single number applied to every protocol, or the per-protocol form `s3=9000,hdfs=9870`.
- The listener binds `127.0.0.1` by default. `--gateway-no-loopback` binds `0.0.0.0` and requires `--gateway-cert` and `--gateway-key`. On loopback those two flags are optional and turn on HTTPS.
- S3 auth is AWS SigV4 (service `s3`) and WebHDFS auth is mountOS SigV4 (service `hdfs`), both using the appserv-issued access key pair the client already holds.
- Each running gateway publishes a descriptor JSON (endpoint URLs, volume, fork, pid) under `~/.mountOS/gateway/`, with a copy in the system temp directory, so local tools can discover the assigned ports.
- The gateway serves general volumes only. On an Iceberg volume the `--gateway` flag is ignored and the client instead starts the Iceberg REST catalog (default port `18181`) and the Iceberg S3 listener (default port `18282`) automatically.

## Kubernetes CSI node driver

`mountos kubernetes` (alias `mountos csi`) is the Kubernetes CSI node driver, Linux only, registered under the driver name `csi.mountos.io`. It serves the CSI Identity, Controller, and Node gRPC services on a unix socket and launches one `mountos` mount subprocess per staged volume, which is what makes a wedged mount survivable: each mount is stopped with its own signal, recovered by pid after a driver restart, and hot upgraded by handing its FUSE file descriptor to a replacement process. Volume attributes on the PersistentVolume are forwarded to the spawned mount as CLI flags, so new mount options work without a driver change.

Flags and defaults:

- `--endpoint` CSI gRPC socket, `/csi/csi.sock`
- `--node-id` Kubernetes node name, defaults to the hostname
- `--mount-bin` client binary launched per volume, defaults to the driver's own executable
- `--state-dir` crash-recovery state, `/var/lib/kubelet/plugins/csi.mountos.io`
- `--health-port` `/healthz` HTTP port, `9810`

No image is published. Build and push from `deploy/csi/Dockerfile` (its entrypoint is `mountos kubernetes`) and pin a version tag in the DaemonSet.

## Desktop app

`mountos-desktop` (`mountOS Desktop`, bundle identifier `sh.mountos.desktop`) is a Tauri 2 + Svelte 5 + TypeScript desktop app for macOS and Windows. It is a supervisor over the public `mountos` CLI, not a mount backend: it re-implements zero protocols. Every action shells out to the documented CLI surface (`mount`, `unmount`, `check --json`, and similar); the UI always shows the exact command it runs. It manages saved mount profiles, stores credentials in the OS keychain/credential vault, launches and monitors mounts, surfaces backend readiness, and produces diagnostics bundles. A main window plus a separate always-on-top tray-popover window are its two surfaces.

It is versioned independently of `VERSION` (its own `tauri.conf.json`/`package.json` version, currently `0.2.0`) and ships as an independently selectable component inside the platform installers: `mountOS Desktop.app` in the `mountos-macos` `.pkg` (alongside FSKit), and its own component in the `mountos-windows` `.exe` bundle (see Download & Install).

### Building and customizing mountOS Desktop

Everything runs through the repo's `Makefile` (`make help` lists all targets):

- `make install` then `make desktop-dev` for local development; `make dev` runs just the browser-facing UI without the Tauri shell.
- `make verify` runs Svelte/TypeScript checks, Rust unit and doc tests, Clippy (`-D warnings`-equivalent strict lint), and the production web build, short of packaging.
- `make bundle` builds distributable bundles: a macOS universal `.app` + `.dmg` (Apple Silicon and Intel in one binary), and, since Windows has no universal format, both `x86_64` and `aarch64` NSIS installers.
- Signing and notarization read credentials only from the environment, never from files in the repo: macOS needs `APPLE_SIGNING_IDENTITY` (bundle-time signing) plus `APPLE_ID`/`APPLE_PASSWORD`/`APPLE_TEAM_ID` for `notarize-macos`; Windows needs `WINDOWS_CERT_THUMBPRINT` and a `TIMESTAMP_URL` RFC 3161 timestamp server, with the EV certificate's private key living on a hardware token (signing is attended unless the token client's PIN caching is enabled). `make release-macos` and `make release-windows` chain bundle, sign, and (on macOS) notarize.
- `make bump-patch` / `bump-minor` / `bump-major` / `set-version VERSION=x.y.z` update the app's own version across `package.json` and `tauri.conf.json` together.

## SDKs and admin clients

| Component / repo | What it is | When you need it |
| --- | --- | --- |
| `mountos-admin-sdk` | Admin SDK for the mountOS Admin API. Wraps the appserv Admin API (`/api/v1/*`) with ED25519 JWT auth. Ships TypeScript (`@mountos-io/admin-sdk`), Go (`github.com/mountos-io/mountos-admin-sdk/go`), and Rust (`mountos-admin-sdk`) packages. Includes a language-neutral API reference generated from `api.yaml`. | Programmatic admin automation (managing users, volumes, teams, keys). |
| `mountos-admin-client` | SvelteKit admin dashboard for mountOS. A static SPA served by a small Bun-based gateway that terminates an Ed25519 auth chain (operator ephemeral JWT, opaque dashboard session token, service JWT to appserv) and reverse-proxies the Admin API. | Web UI for administering a mountOS deployment. |
| `mountos-hdfs-sdk` (`hadoop-mountos`) | Hadoop Compatible FileSystem (HCFS) jar providing `mountos://` URI support. Wraps Hadoop's `WebHdfsFileSystem` and injects AWS SigV4 (service `hdfs`) using the access key/secret in `core-site.xml`. | Spark/Hive/Trino/distcp/Flink access to mountOS via stock Hadoop tooling, pointed at `mountos gateway --gateway hdfs`. |

### Admin API resources

The Admin API (consumed via `mountos-admin-sdk` or `mountos-admin-client`) covers resources including Accounts, Users, Regions, RegionClusters, Storages, Volumes, VolumeForkTrees, VolumeForkEntries, VolumeForkSearches, AuditLogs, RegionAuditLogs, ServiceNodes, Nodes, ClientSessions, Discover, Dashboard, License, Alerts, RegionAlerts, and Vault. License is a managed resource: the HUB holds the active license payloads in its admin database, and the API uploads and lists them. Stored licenses are append-only and leave the active set only by expiry. Licenses of one identity stack, so their capacities add and a renewal extends coverage. The HUB serves the Free tier when the stored set spans more than one identity.

### Admin dashboard (mountos-admin-client) walkthrough

The admin dashboard is a SvelteKit single-page app that an operator uses to administer a mountOS deployment through the same Admin API the SDK targets. It is built as a static bundle (`@sveltejs/adapter-static`, SPA fallback `index.html`) and served at runtime by a small Hono server on Bun, which also terminates authentication and reverse-proxies the Admin API to appserv.

#### Operator areas

The left navigation exposes these areas (each gated by a feature flag and, for some, an admin-only flag):

- Dashboard, an overview/landing view of the deployment.
- Accounts, create and manage accounts (including account lock).
- Users, create and manage users.
- Regions (admin-only), manage regions and, within a region, its RegionClusters, Nodes, and a topology view.
- Storages (admin-only), manage storage backends.
- Volumes, create and manage volumes. Per-volume operations include browsing the fork tree, generating and revoking volume access keys (the appserv-issued S3 key pairs), setting quota, and viewing volume stats. A per-account "user" role can self-service its own volume API keys.
- Audit Log, browse audit events.
- Sessions, view and manage client sessions.
- Nodes, view service nodes and their health.
- Alerts, view deployment alerts.

Team membership and Vault are resources the appserv backend exposes through the Admin API; they are managed through the relevant account/volume and admin flows rather than as separate top-level nav entries. License is managed under Settings then License (admin only), where an operator uploads license payloads and lists the active set. Licenses are append-only and leave the active set only by expiry, and same-identity licenses stack their capacity. (Whether each of these surfaces is exposed in the GA build is a product decision and not determinable from the codebase.)

#### Deploy and run

The dashboard ships as the static build plus the Bun gateway. The gateway serves the built SPA, sets and recovers httpOnly `Strict`-SameSite session and refresh cookies, applies CSRF protection, Redis-backed rate limits, and (optionally) WebAuthn passkey step-up, then reverse-proxies `/api/v1/*` to appserv.

- Build the SPA with the project's `build` target (Vite build to a static bundle).
- Run the gateway as a Bun process. It listens on `PORT` (default `3001`) and forwards to `MOUNTOS_APPSERV_URL`.
- For local development, the project's `dev` target runs the Vite dev server (default `:5173`) and `dev-all` additionally runs the Bun gateway, so the SPA and the auth/proxy layer run together.

Required environment for the gateway (the server refuses to start if any are missing):

- `PROVIDER2DASHBOARD_VERIFICATION_KEY`, the Ed25519 public key used to verify the operator's ephemeral login tokens.
- `DASHBOARD_SIGNING_KEY` and `DASHBOARD_VERIFICATION_KEY`, the Ed25519 private seed and public key used to sign and verify the dashboard's own session and refresh tokens.
- `MOUNTOS_PRIVATE_KEY`, the Ed25519 private seed used to mint the service JWT to appserv.
- `MOUNTOS_APPSERV_URL`, the appserv base URL the gateway proxies to.
- `REDIS_URL`, the Redis instance backing sessions, rate limiting, and WebAuthn state.

#### Ed25519 JWT auth chain

Authentication is a three-hop Ed25519 JWT chain, terminated in the Bun gateway so the browser never holds long-lived service credentials:

1. Operator to dashboard. The operator's backend mints a short-lived ephemeral EdDSA (Ed25519) JWT for the user (`aud=mountos/dashboard`, `sub` the user id, a `role`, plus `username` and `account_id` for the restricted `user` role), signed with the private seed whose public half the dashboard holds as `PROVIDER2DASHBOARD_VERIFICATION_KEY`, and redirects to `https://<dashboard>/?token=<jwt>`. Tokens are minted roughly 60 seconds out; the dashboard verifies at `POST /api/auth/exchange` and rejects any token whose issue time is more than 120 seconds old (60 seconds of clock leeway). The Admin SDK topic documents the full claim set.
2. Dashboard session. On a valid operator token, the gateway issues an opaque session token plus a refresh token (backed by the `DASHBOARD_SIGNING_KEY` / `DASHBOARD_VERIFICATION_KEY` pair) and stores them as httpOnly cookies. The browser presents the session token on subsequent calls; expired sessions are silently refreshed from the refresh cookie, and sessions can be revoked.
3. Service JWT to appserv. For proxied `/api/v1/*` requests, the gateway attaches a service JWT minted and cached by the admin SDK's token signer (1 hour TTL, refreshed 5 minutes before expiry; keyed by `MOUNTOS_PRIVATE_KEY`) and a separately signed `X-MountOS-Dashboard-User` header carrying the caller's identity and role. appserv verifies both and enforces role-scoped authorization, so the dashboard cannot act outside the caller's permissions even if a client is tampered with.

The gateway adds further hardening on top of this chain: WebAuthn (passkey) step-up for sensitive operations, per-route rate limits, CSRF protection, and server-side enforcement of role constraints (for example, blocking restricted volume fields and pinning per-user data scoping at the proxy) so the same rules hold whether a request comes from the UI or a direct API call.

### hadoop-mountos build and configuration

The jar is published under maven coordinates `io.mountos:hadoop-mountos`. It is built with two profiles: `hadoop3` (built against Hadoop 3.4.1, jar classifier `hadoop3`) and `hadoop2` (built against Hadoop 2.10.2, jar classifier `hadoop2`). The Java SPI registers `io.mountos.hadoop.MountOSFileSystem` for the `mountos` URI scheme.

Configuration goes in `core-site.xml` (or as `spark.hadoop.*` overrides). The relevant keys are:

- `fs.mountos.impl` = `io.mountos.hadoop.MountOSFileSystem`
- `fs.AbstractFileSystem.mountos.impl` = `io.mountos.hadoop.MountOSAbstractFileSystem`
- `fs.mountos.endpoint` (the WebHDFS endpoint published by `mountos gateway --gateway hdfs`)
- `fs.mountos.access.key` and `fs.mountos.secret.key` (the appserv-issued S3 access key pair used to sign requests)
- `fs.mountos.fork` = `<name>` to select a named fork

## How the pieces fit

- Filesystem access: an OS mount client (`mountos` on Linux, FSKit/FUSE/NFS/SMB on macOS, the mountosio kernel driver on Windows) connects to `dataserv` for metadata (TCP, default 6464) and, where the optional `blockserv` is deployed, to `blockserv` for byte I/O (block protocol, `BLOCK_PORT` default 9100).
- Object/protocol access: `mountos gateway --gateway s3,hdfs` fronts the same volume over S3 REST and WebHDFS on a local endpoint, using the SigV4 keys issued by `appserv`.
- Control/admin: `appserv` is the auth and management plane (the HUB); automate it with `mountos-admin-sdk` or use the `mountos-admin-client` dashboard, whose Bun gateway terminates the Ed25519 JWT chain and proxies the Admin API.
- Maintenance: `gcserv` runs cleanup, repair, retention, and compaction in the background.
- Kubernetes: `mountos kubernetes` (driver `csi.mountos.io`) provisions volumes for pods by launching one mount per volume.


<!-- topic: deployment -->

# Deployment

mountOS is a software suite the operator self-hosts. It ships as **binaries**, pulled per
service from `download.mountos.sh`. There are no published container images and no build from
source on the operator path. A deployment is one global **HUB** plus, per region, a set of
cluster-scoped services, wired to a small set of external dependencies the operator already
runs or provisions (a database, a secret store, and DNS).

This page describes the deployment surface. The step-by-step bring-up is the provision skill.
The full config surface is the configuration topic, and the secret layout is the secrets topic.

## Service fleet

| Binary | Role | Scope | State | Public IP | DNS | TLS |
| --- | --- | --- | --- | --- | --- | --- |
| `appserv` | HUB: Admin API, discovery, registration, licensing | one per deployment (replicate for HA) | none (admin DB) | yes | yes | yes (HTTPS) |
| `dataserv` | Metadata backbone, Raft owner per `(volume, fork)` | per cluster, quorum of 3 | Raft data dir + mmap arena | yes | no | no (custom protocol/TCP) |
| `gcserv` | Garbage collection, compaction, key rotation | per cluster | none (shares region DB) | only if standalone | no | no |
| `blockserv` (optional) | Block-storage byte plane (block protocol), alternative to a cloud object backend | up to 3 active-active block volumes per storage, across distinct clusters | local disk cache (preferably XFS) that syncs to the backing store | yes | no | no (direct IP) |
| `mountos` | Client: FUSE/NFS/SMB/FSKit mount, embedded S3/WebHDFS gateway, Kubernetes CSI node driver | per host/pod | local cache | n/a | n/a | n/a |

Four server services, one client binary. S3, WebHDFS, and Kubernetes CSI are modes of the
client, not services to deploy. There are no `gatewayserv`, `indexnotifyserv`, `leaseserv`,
`webserv`, or `otelserv` services.

## External dependencies

| Dependency | Scope | Notes |
| --- | --- | --- |
| Admin database | global, on the HUB | PostgreSQL or MySQL. `appserv db install` creates the schema. |
| Region database | per region | PostgreSQL or MySQL. `dataserv db install` creates the schema; `gcserv` shares it. |
| Secret store (vault) | one per region, plus the Hub vault | HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. |
| DNS | the HUB | The HUB needs a stable domain. `dataserv`, `gcserv`, and `blockserv` need none. |

Object storage is **not** a deployment artifact. A `storage` is a control-plane record that
points at an existing S3-compatible or Azure store (AWS S3 and Cloudflare R2 work as is),
created at provisioning time, not stood up by a deploy template. There are two dialects,
PostgreSQL (preferred) and MySQL (any MySQL-wire store). Use the PostgreSQL dialect unless you
already run a MySQL-wire store; it batches more efficiently, and the schema ships for both, so
the choice costs no capability.

## Acquire the binaries

mountOS ships binaries, not images. Pull each service, pinning a version:

```sh
curl -fsSL https://download.mountos.sh | bash -s -- --pkg mountos-appserv --version X.Y.Z
# region services:
curl -fsSL https://download.mountos.sh | bash -s -- --pkg mountos-dataserv --version X.Y.Z
curl -fsSL https://download.mountos.sh | bash -s -- --pkg mountos-gcserv   --version X.Y.Z
# optional: mountos-blockserv
# client: mountos-cli
curl -fsSL https://download.mountos.sh | bash -s -- --list   # list packages
```

The installer resolves the right artifact for the host platform and verifies its checksum.
Each binary prints its own config template with `<binary> env` (stdout) or `<binary> env -w .env`
(file); prefer that over copying samples, it matches the running build exactly.

## Bring-up order

The order is load-bearing; a skipped step surfaces in the next one.

1. **HUB.** Run `appserv` with its admin database and Hub vault. Set `DB_URL` and `DB_DIALECT`
   in the vault, run `appserv db install`, and publish `appserv` behind a stable TLS domain
   (`<HUB>`). At first bootstrap appserv creates the reserved hub region and its default `uno`
   cluster together, so the hub matches the every-region-has-a-uno model. Later upgrades use
   `appserv db migrate`.
2. **Provision.** Through the Admin API (open-source admin SDK, or the admin client): create an
   account, its users, and a region. A region auto-creates its default cluster `uno`, which
   accepts no volumes until a cluster-scoped service registers into it.
3. **Region services.** Provision the region database and Region vault, run `dataserv db install`,
   then run `dataserv` (3 nodes for quorum) and `gcserv`. They self-register with the HUB. The
   first registration flips `uno` to ready.
4. **Volumes and mount.** Create a storage, a volume, and an access key, then mount.

## Network model

- `appserv` always has a public IP, a DNS domain, and TLS (HTTPS for the Admin API and
  discovery).
- `dataserv` advertises a public IPv4 (`ADVERTISE_ADDR`, an IP, not a hostname), speaks the
  custom protocol over TCP with no TLS and no DNS. Ports: `APP_PORT` (default `6464`), `RAFT_PORT` (`6465`),
  `RPC_PORT` (`6466`); keep them fixed across restarts.
- `gcserv` needs no DNS and stays hidden inside the region; it needs a public IP only when it
  runs on its own instance (so the HUB can reach it), not when co-located on a `dataserv` node.
- `blockserv` advertises a public IP and is reached by direct IP, no TLS (a transport
  revamp is pending).
- The embedded S3/WebHDFS gateway is a client-side listener. It binds `127.0.0.1` unless
  `--gateway-no-loopback` is passed, which requires a TLS cert and key. Nothing about it is a
  fleet concern.

Clients never need to know their region. They resolve through the HUB's unauthenticated
discovery endpoint (`GET /api/v1/discover/meta?accessKeyId=<id>`), which returns the owning
cluster's `dataserv` endpoints; the result is cached briefly at the HUB and on the client.

## High availability

Run `dataserv` as a Raft quorum, 3 nodes per cluster (5 for larger clusters; odd counts only,
more only adds quorum overhead). Replicate `appserv`, and use a highly-available database and
secret store. The first `dataserv` registration marks the cluster ready; quorum keeps it
serving through node loss.

## Secrets

Each service reads its keys from a per-service vault path (`mountos/<service>`, for example
`mountos/appserv`, `mountos/dataserv`; a deployment-wide `VAULT_RESOURCE_PREFIX` renames the
root to `mountos-<prefix>/<service>`, see the secrets topic). The operator provides the admin
Ed25519 key pair
(keep the private seed; seed only the public key into `appserv` as `PROVIDER_VERIFICATION_KEY`).
`appserv` also needs a dedicated `DASHBOARD_USER_HMAC_KEY` (a random secret, distinct from the
public verification key) shared only with the dashboard backend that signs the
`X-MountOS-Dashboard-User` header. The `service-verifiers` set is replicated from the Hub vault
to every Region vault; cross-service JWT verification fails without it. The full layout is in
the secrets topic.

Licenses live in the admin database on the HUB, not in files on each server. The HUB serves the
current license state to every service over the 30 second heartbeat. Operators manage licenses
through the admin client or the admin SDK (upload, list), not by placing files. appserv
may optionally take `MOUNTOS_LICENSE_PATH` once to seed the first license into the admin database on
boot. With no active license the deployment runs on the Free tier, up to 10 TiB.

## What ships today

Under `mountos-servers/deploy/`:

- `csi/`: the Kubernetes CSI node driver manifests and image build (client side, mounts volumes into pods).
- `systemd/`: host mount units for the `mountos` client (client side).

Server-fleet deploy templates (Compose, Helm, Terraform, Ansible) and the bring-up bootstrap
are being added under `deploy/`; see `deploy/README.md` for status. Until then, the server
bring-up follows the provision skill against binaries pulled from `download.mountos.sh`.

## Kubernetes CSI driver

`mountos kubernetes` registers as `csi.mountos.io` and exposes mountOS volumes as
PersistentVolumes (ReadWriteMany, over FUSE). It runs as the `mountos-csi-node` DaemonSet in
`mountos-system`, launching one `mountos` mount subprocess per volume. The `CSIDriver` object
sets `attachRequired: false`.

A PersistentVolume points `discoveryURL` at the **appserv HTTPS discovery URL** (the `<HUB>`
domain), carries the 20-character access-key id as `volumeHandle`, and supplies the key pair
through `nodeStageSecretRef` (Secret keys `accessKeyID` and `secretAccessKey`). The driver
launches the FUSE client, which resolves `dataserv` through appserv discovery, so the attribute
is the appserv HTTPS URL, not a raw `dataserv` `host:port`. The driver routes `discoveryURL` to
the client's `MOUNTOS_DISCOVERY_URL`; any other volume attribute passes through as a client flag
(`xattr`, `acl`, `mount-opts`, `disk-cache-dir`, and the rest). Driver health is HTTP `/healthz`
on port `9810`. No CSI image is published; build and push from `deploy/csi/Dockerfile` (its
entrypoint is `mountos kubernetes`) and pin a version tag in the DaemonSet. See `deploy/csi/`
for the manifests.

## Single-host (systemd)

For a host mount of the `mountos` client, `deploy/systemd/` ships a templated `mountos@.service`
(`Type=forking`, reads `/etc/mountos/%i.env`, mounts on start and unmounts on stop) and a native
`.mount` unit that requires the mount helper (`mountos mount --install-mount-helper`, which
installs `/sbin/mount.mountos`). These mount an existing volume; they do not stand up server
services.

## Read next

- Bring-up steps: the provision skill.
- Day-2 operations: the operate skill.
- Config surface: the configuration topic.
- Secret layout: the secrets topic.
- Installer and downloads: https://download.mountos.sh


<!-- topic: configuration -->

# Configuration and Environment

mountOS server components are configured almost entirely through environment variables (typically loaded from a per-component `.env` file), with a small set of build-time identity variables baked into binaries and a richer command-line flag surface on the `mountos` client. This page describes the configuration surface a deployer sets, grouped by component, with defaults and required status where confirmed from the shipped templates and binary code.

## Naming convention

All mountOS-defined environment variables use the `MOUNTOS_` prefix (e.g. `MOUNTOS_ACCESS_KEY_ID`, `MOUNTOS_DISCOVERY_URL`, `MOUNTOS_LICENSE_PATH`). Variables without that prefix (`VAULT_*`, `SERVICE_RPC_ADDR`, `REGION_CLUSTER_ID`, `PORT`, etc.) belong to a distinct config surface (secret-store backend, service topology) and are documented per component below. Internal, hidden tuning knobs use a leading and trailing underscore (`_MOUNTOS_..._`) to mark them as not intended for normal operator use.

## Components

mountOS ships four server binaries, each built with Go build tags (`defaultserv && <service>`), plus the client.

| Binary | Role |
| --- | --- |
| `appserv` | Central hub. Service discovery, registration, heartbeat, licensing, rate limiting. |
| `dataserv` | Metadata server (in-memory arena cache backed by DB, Raft ownership). |
| `blockserv` | Block storage server (hybrid local cache with cache-to-S3 sync). |
| `gcserv` | Garbage collection and background work goals (retention, compaction, key rotation). |
| `mountos` | FUSE/NFS/SMB/FSKit client. Also runs the embedded S3/WebHDFS gateway and the Kubernetes CSI node driver. |

The server binaries read their configuration from environment variables (sourced from a `.env` file, a container/helm-injected environment, or the secret store), and redacted `.env-sample` templates are generated from each service's `.env` (see Deployment-level variables). The client is configured by CLI flags plus a small set of `MOUNTOS_*` variables.

## Build-time identity variables

The build injects identity values via `-ldflags -X` (not runtime env). The operator-facing Make inputs are:

| Make variable | Default | Effect |
| --- | --- | --- |
| `BUILD_CLIENT_ID` | `mountOS` | Sets `identity.ClientID` (tenant brand baked into the binary). |
| `BUILD_PROVIDER_DOMAIN` | `mountos.io` | Sets `identity.Provider`. |
| `BUILD_CLIENT_DOMAIN` | `mountos.io` | Sets `identity.ClientDomain`. |

The build also injects `ServiceName`, `BinaryName`, `BuildTime`, `CommitHash`, and `Version` at compile time. `Version` and `CommitHash` are reported by `--version` (formatted as `Version (CommitHash)`).

## Common environment variables

These appear across most or all server components (exact presence varies, see component sections).

### Runtime and identity

| Variable | Meaning | Required |
| --- | --- | --- |
| `MOUNTOS_LICENSE_PATH` | Licenses are delivered by the HUB from the admin database over the heartbeat, so services take no license path. appserv only: an optional one-time bootstrap seed. If set, appserv loads that file into the admin `licenses` table on boot when no matching payload is present, even if expired, then the database is the source of truth. Region services ignore it. | No |
| `SERVICE_RPC_ADDR` | Appserv internal RPC address for registration, heartbeat, and discovery. Format `host:port`. | Yes (server components) |
| `REGION_CLUSTER_ID` | Region cluster UUID. Accepts dashed or compact UUID. Must reference an active `region_clusters` row. The cluster carries its owning region, so the region shard (and account) are derived from it via appserv at startup (no separate `REGION_ID`). A deactivated cluster raises a critical topology alert and the service refuses to register. appserv (the hub) carries it for env-shape consistency only; it resolves its own reserved hub `uno` cluster internally and does not consume this value. | Yes (regional services) |

### Networking and discovery

| Variable | Meaning | Default |
| --- | --- | --- |
| `ADVERTISE_ADDR` | Explicit IPv4 advertise address (hostname/DNS not allowed). Localhost OK for local dev, public IP required in production. | Auto-detected if unset |
| `CLOUD_PROVIDER` | IMDS hint for public IP discovery (`aws`, `gcp`, `azure`, `digitalocean`, `oci`, `hetzner`). | Auto-probe all providers if unset |

### Secret store (Vault)

mountOS reads signing keys and DB credentials from a secret store. The provider is selectable.

| Variable | Meaning |
| --- | --- |
| `VAULT_PROVIDER` | Secret store provider. Supported: `hashicorp`, `aws`, `gcp`, `azure`. |
| `VAULT_HASHICORP_ADDRESS`, `VAULT_HASHICORP_ROLE_ID`, `VAULT_HASHICORP_SECRET_ID` | HashiCorp Vault AppRole connection parameters. |
| `VAULT_AWS_REGION`, `VAULT_AWS_ACCESS_KEY_ID`, `VAULT_AWS_SECRET_ACCESS_KEY` | AWS Secrets Manager. Optional, standard AWS credential chain (env, IAM role, instance profile) used otherwise. |
| `VAULT_GCP_PROJECT_ID`, `VAULT_GCP_CREDENTIALS_FILE` | GCP Secret Manager. Optional, Application Default Credentials used otherwise. |
| `VAULT_AZURE_URL`, `VAULT_AZURE_TENANT_ID`, `VAULT_AZURE_CLIENT_ID`, `VAULT_AZURE_CLIENT_SECRET` | Azure Key Vault. Optional, `DefaultAzureCredential` used otherwise. |
| `VAULT_RESOURCE_PREFIX` | Optional, empty by default. Namespaces every secret name under `mountos-<prefix>` instead of the bare `mountos` root, so multiple deployments can share one cloud account without colliding. Must match across every service in the deployment. |
| `ED25519_SIGNING_KEY`, `ED25519_VERIFICATION_KEY` | Ed25519 signing/verification keys, normally sourced from the secret store under a per-service vault path. |

Per-service vault paths follow `mountos/<service-name>`: `mountos/appserv`, `mountos/dataserv`, `mountos/gcserv`, and `mountos/blockserv` (or `mountos-<prefix>/<service-name>` when `VAULT_RESOURCE_PREFIX` is set). The client has no vault path; it authenticates with an access key pair.

### Database (appserv, dataserv, gcserv)

The DB connection values (`DB_DIALECT`, `DB_PROVIDER`, `DB_URL`, `DB_PROVIDER_VERSION`) are typically supplied through the secret store, with `DB_URL` and `DB_PROVIDER_VERSION` also accepted as plain env keys.

| Variable | Meaning | Default |
| --- | --- | --- |
| `DB_DIALECT` | `mysql` or `postgresql`. Sourced from config/vault. | (required where DB is used) |
| `DB_PROVIDER` | Specific provider within the dialect, used to select the capability profile. MySQL dialect: `mysql`, `mariadb`, `tidb`, `singlestore`, `aurora-mysql`, `vitess`. PostgreSQL dialect: `postgresql`, `cockroachdb`, `yugabytedb`, `alloydb`, `aurora-postgresql`. An unknown provider fails fast at startup. | `mysql` if unset |
| `DB_URL` | Database connection URL (sourced from vault). | (required) |
| `DB_PROVIDER_VERSION` | Provider version string for capability detection. Parsed generically (a leading semantic version such as `15`, `16`, `8.0.32`) and compared against per-provider, per-capability minimums. It is not validated against a fixed allow-list. | (optional, sane fallbacks used) |
| `DB_MAX_OPEN_CONNS` | Connection pool sizing. Defaults to a per-CPU baseline, scaled up on distributed SQL engines (CockroachDB, YugabyteDB, TiDB, Vitess, SingleStore). Max-idle is not configurable and always tracks this value. | optional |
| `DB_CONN_MAX_LIFETIME_IN_MINS`, `DB_CONN_MAX_IDLE_TIME_IN_MINS` | Connection lifetime/idle caps in minutes. | optional |
| `DB_POOL_MIN_CONNS` | Minimum warm connections (avoids cold-connection penalty). | optional |
| `DB_POOL_HEALTH_CHECK_SECS` | Pool replenish health-check interval (seconds). | optional |

`DB_PROVIDER_VERSION` drives feature gating rather than acting as a compatibility allow-list. When the version is omitted, mature features default to enabled where it is safe (for example PostgreSQL `SKIP LOCKED` defaults on because all supported releases include it). Representative capability thresholds:

| Capability | Gate |
| --- | --- |
| `SELECT ... FOR UPDATE SKIP LOCKED` | MySQL/Aurora MySQL `8.0.1+`, MariaDB `10.6+`, PostgreSQL `9.5+` (default on when version unknown), CockroachDB `22.1+`, YugabyteDB `2.13+`; not used on TiDB/SingleStore/Vitess. |
| `FOR UPDATE OF <alias>` | MySQL/Aurora MySQL `8.0+`, all PostgreSQL-dialect providers; not MariaDB/TiDB/SingleStore/Vitess. |
| Window functions | MySQL `8.0+`, MariaDB `10.2+`, all PostgreSQL-dialect providers, TiDB, SingleStore; not Vitess. |
| `JOIN LATERAL` | MySQL/Aurora MySQL `8.0.14+` (default on when version unknown), all PostgreSQL-dialect providers; not MariaDB/TiDB/SingleStore/Vitess. |

### Observability

Metrics auth has no env var: `/metrics` on every service requires a bearer JWT minted by appserv (`POST /api/v1/metrics/token`, admin-only, optional `expirySeconds` -- 0 never expires), scoped to the `mountos/metrics` audience so it can't be reused as any other kind of token. Verification reuses each service's existing service-identity trust fabric (the same mechanism internal RPC peer auth already relies on) to check the signature against appserv's public key -- no shared secret to distribute or rotate by hand.

| Variable | Meaning | Default |
| --- | --- | --- |
| `PROMETHEUS_ENDPOINT_SUFFIX` | Custom suffix for the Prometheus metrics path. Endpoint becomes `/metrics-{suffix}`, otherwise an auto-generated hashed suffix derived from the build identity. | auto-generated |
| `METRICS_PORT` | Dedicated metrics listener port. blockserv defaults to `8082`; unset elsewhere means no separate listener and metrics served on the main HTTP surface. The dedicated path is `/metrics` or `/metrics-{PROMETHEUS_ENDPOINT_SUFFIX}`. | `8082` (blockserv); unset otherwise |

## Component-specific configuration

A note on `PORT`. The non-TLS server components (dataserv, gcserv) expose a generic HTTP surface. When no TLS cert/key/`HTTPS_PORT` is configured, the HTTP listener binds the `PORT` value and defaults to `8080` when `PORT` is empty. The same `PORT` value also feeds internal RPC self-registration on services that register a TCP endpoint; there the registration default is `0` (no TCP registration advertised unless `PORT` is set), while the HTTP server still falls back to `8080`.

### appserv

| Variable | Meaning | Default |
| --- | --- | --- |
| `HTTPS_PORT` | HTTPS listener port. | (required) |
| `TLS_CERT_FILE`, `TLS_KEY_FILE` | TLS certificate and key paths. | (required) |
| `RATE_LIMIT_IP_REQUESTS_PER_SECOND` | Per-IP average RPS for unauthenticated requests. | `1000` |
| `RATE_LIMIT_IP_BURST_PERCENT` | Burst capacity as a percentage of IP RPS. | `25` |
| `RATE_LIMIT_USER_REQUESTS_PER_SECOND` | Per-user average RPS for authenticated requests. | `20` |
| `RATE_LIMIT_USER_BURST_PERCENT` | Burst capacity as a percentage of user RPS. | `50` |
| `ALERT_RETENTION_DAYS` | Days to retain resolved alerts before cleanup. | `28` (min 28) |
| `MOUNTOS_STORAGE_ALLOW_PRIVATE` | Allow storage-credential validation to dial loopback/link-local/private endpoints. Leave unset for cloud backends; set to `1` only when the object store legitimately lives on a private network (for example an in-cluster MinIO). | `false` |

### dataserv

| Variable | Meaning | Default |
| --- | --- | --- |
| `APP_PORT` | Primary TCP port. | `6464` |
| `RAFT_PORT` | Stable Raft transport port. Must stay fixed across restarts. | `APP_PORT+1` (`6465`) |
| `RPC_PORT` | Stable internal RPC port for discovery and peer RPCs. | `APP_PORT+2` (`6466`) |
| `RAFT_DATA_DIR` | Base path for Raft data. The generated raft node ID is appended so each local instance gets its own directory. | `/var/lib/mountos/raft` |
| `PORT` | HTTP server port (non-TLS surface). | `8080` |
| `METAENGINE_ARENA_SIZE` | Total mmap'd memory for the in-memory metadata cache, shared across shards, allocated upfront at start. A slot is 192 bytes plus a name allowance, so plan about five million records per GiB; names over the allowance push memory up a bit. Format `<number><MB|GB>`, min `128MB`. | (required) |
| `SEM_PER_CONNECTION` | Per-connection local dispatch semaphore capacity. | `8` (min 8) |
| `GLOBAL_SEM_FOR_CONNECTIONS` | Global overflow dispatch semaphore. `0` auto-computes `max(NumCPU*NumCPU, 250)`. | `0` (min 250 when nonzero) |
| `TCP_BACKPRESSURE_ENABLED` | Enable TCP backpressure control. | `true` |
| `TCP_MAX_CONNECTIONS` | Max concurrent TCP connections. 0/unset = adaptive (derived from measured DB latency, scales down from 10000; past 105% of the ceiling new connections are rejected). Positive value (range 1-1000000) pins a static ceiling. | `0` (adaptive) |
| `TCP_RATE_LIMIT_REQUESTS_PER_SECOND` | Per-connection TCP request rate. 0/unset = adaptive (same latency gradient, scales down from 1000). Positive value (range 1-100000) pins a static rate. | `0` (adaptive) |
| `TCP_RATE_LIMIT_BURST_PERCENT` | Burst allowance as percent of RPS (range 0-1000). | `25` |
| `TCP_GRACEFUL_DELAY_MAX_MS`, `TCP_GRACEFUL_DELAY_STEP_MS` | Graceful degradation max delay and step (ms, range 0-60000). | `500` / `50` |
| `CYCLE_DETECTION` | Strict POSIX rename cycle detection (walk full parent chain). `false` checks only direct self-reference and immediate parent. | `false` |
| `PARENT_CMTIME_SKIP_TTL_MS` | TTL for skipping redundant parent dir mtime/ctime updates during bursts (max 5000, `<=0` disables). | `999` |
| `EXPERIMENTAL_MSG_ZEROCOPY` | Experimental TCP `MSG_ZEROCOPY` for large sends (>10KB) on Linux 4.14+. Skipped for TLS and loopback. | `false` |

The default TCP port (`6464`) is the package constant `DefaultTCPPort`. `RAFT_PORT` and `RPC_PORT` derive from `APP_PORT`, so a single override shifts the whole triple while keeping the offsets fixed.

### blockserv

| Variable | Meaning | Default |
| --- | --- | --- |
| `BLOCK_VOLUME_ID` | This block volume's stable identity, the appserv fetch key used to resolve its storage at startup. Each block volume of a storage has its own. | (required) |
| `BLOCK_PORT` | Client byte I/O data-plane port (custom, encrypted block protocol). Peer replication binds `BLOCK_PORT`+1. | `9100` |
| `STORAGE_MOUNT_ROOT_PATH` | Storage mount path root (absolute, existing, writable directory). Format the mounted block device preferably with XFS and mount it `rw,async,noatime,nodev,noexec,nosuid` (writable async cache, atime off, and `nodev`/`noexec`/`nosuid` hardening for a data-only blob mount; XFS uses `inode64` by default). | (required) |
| `DELETE_MODE` | Reclaimed-block deletion behavior. `normal` (default) is a plain unlink (data may be forensically recoverable). `secured`/`secured-immediate` both zero-overwrite the block (with `fsync`) before unlink, inline at the block-storage layer, for compliance. Invalid or empty values fall back to `normal`. | `normal` |
| `CACHE_SYNC_SWEEP_INTERVAL` | Cache-to-S3 sync worker sweep interval. | `5s` |
| `CACHE_SYNC_LEASE_TTL`, `CACHE_SYNC_LEASE_RENEW` | Claimed-job lease TTL and renewal interval. | `120s` / `40s` |
| `CACHE_SYNC_MAX_UPLOADS` | Max concurrent S3 uploads. | `4` |
| `CACHE_INLINE_S3_MAX` | Advisory ceiling for concurrent inline degraded/bypass S3 PUTs; the gate self-tunes below this on S3 backpressure. `0` reuses `CACHE_SYNC_MAX_UPLOADS`. | `0` |
| `CACHE_SYNC_RETRY_MAX`, `CACHE_SYNC_RETRY_BASE` | Upload retry count and backoff base. | `10` / `5s` |
| `CACHE_SYNC_S3_CLIENT_TTL` | S3 client cache TTL. | `15m` |
| `CACHE_SYNC_DEDICATED_VOLUME` | Treat cache as a dedicated volume. | `true` |
| `CACHE_EVICTOR_MAX_SIZE` | Cache evictor max size in bytes. | `1099511627776` (1 TiB) |
| `CACHE_EVICTOR_HIGH_WATERMARK`, `CACHE_EVICTOR_LOW_WATERMARK` | Eviction trigger and target ratios. | `0.85` / `0.75` |
| `CACHE_EVICTOR_MIN_FREE_RATIO` | Min free disk ratio before forced eviction. | `0.10` |
| `CACHE_EVICTOR_CHECK_INTERVAL`, `CACHE_EVICTOR_RECONCILE_INTERVAL` | Eviction check and full reconciliation intervals. | `30s` / `5m` |

### gcserv

| Variable | Meaning | Default |
| --- | --- | --- |
| `PORT` | HTTP server port (non-TLS surface). Also used for internal RPC self-registration when set. | `8080` (HTTP); `0` = no TCP registration |
| `API_KEY_ROTATION_INTERVAL_DAYS` | Days between API key master key rotations. Re-encrypts existing secrets, user-facing secrets unchanged. | `7` (min 3, max 30) |
| `MAX_CONCURRENT_UPLOADS`, `MAX_CONCURRENT_DOWNLOADS` | Segment compaction read/write parallelism overrides. | optional |

### Embedded gateway (mountos)

The S3 and WebHDFS surfaces are driven by `--gateway*` flags on the client, listed with the client flags below. A few WebHDFS behaviours are env-only:

| Variable | Meaning | Default |
| --- | --- | --- |
| `MOUNTOS_HDFS_TRUSTED_HOSTS` | Comma-separated allowlist of public host(s) (`host` or `host:port`) reflected into two-phase WebHDFS redirects (CREATE/APPEND/OPEN). Required to enable those operations. When unset the redirect Host is not trusted and such requests return 421, preventing host-header injection and open redirect. | (empty, redirects disabled) |
| `MOUNTOS_HDFS_FILE_CHECKSUM` | Enable `GETFILECHECKSUM`. When false, returns 501 so Hadoop/distcp skip checksum verification. | `false` |
| `MOUNTOS_HDFS_UPLOAD_TIMEOUT_SEC` | Upload timeout in seconds. A positive value overrides the built-in default. | built-in default |
| `MOUNTOS_HDFS_MAX_CONCURRENT_BLOCK_UPLOADS` | Cap on concurrent block uploads. `0` keeps the built-in default. | `0` |

### mountos client

`mountos` is configured by CLI flags plus a small set of environment variables, chiefly `MOUNTOS_ACCESS_KEY_ID`, `MOUNTOS_SECRET_ACCESS_KEY`, and `MOUNTOS_DISCOVERY_URL`. `MOUNTOS_DISCOVERY_URL` is the appserv **HTTPS URL**, used for the `/api/v1/discover/meta` REST call, not a raw `dataserv` `host:port`.

`mountos kubernetes` launches one `mountos` mount subprocess per volume and passes `MOUNTOS_CSI_MODE=1`, `MOUNTOS_ACCESS_KEY_ID`, `MOUNTOS_SECRET_ACCESS_KEY`, and `MOUNTOS_DISCOVERY_URL` into it. The driver itself is configured by its own flags (`--endpoint`, `--node-id`, `--mount-bin`, `--state-dir`, `--health-port`) rather than an `.env` file.

`mountos` is a cobra application. Key global flags (most also have env equivalents):

| Flag | Env | Meaning | Default |
| --- | --- | --- | --- |
| `--mount, -m` | `FUSE_MOUNT_POINT` | Mount point directory (required for a mount). | |
| `--access-key-id, -a` | `MOUNTOS_ACCESS_KEY_ID` | Access key ID (20 chars). | |
| `--secret-access-key, -s` | `MOUNTOS_SECRET_ACCESS_KEY` | Prompt for / supply secret key (40 chars). | |
| `--discovery-url` | `MOUNTOS_DISCOVERY_URL` | Appserv URL for discovery (required). | |
| `--volname, -n` | | Volume display name. | volume name |
| `--foreground, -f` | | Run in foreground (no daemonize). | false |
| `--debug, -d` | `FUSE_DEBUG` | Verbose debug mode. | false |
| `--mount-opts, -o` | | Comma-separated mount options (FUSE: `allow_other`, `allow_root`, `ro`, `debug`; NFS-style: `root_squash`, `all_squash`, `anonuid=N`, `anongid=N`). | |
| `--mem-limit` | | Soft memory limit (for example `512M`, `2G`), `0M`/`0G` = no limit. | |
| `--mem-arena` | | Metadata arena size (min `20M`), mmap'd outside Go heap. | `100M` |
| `--buffer-size` | | Memory buffer cap (MiB/GiB), `0` = no memory control. | up to 1 GiB, scaled from RAM |
| `--buffer-slack` | | Overflow ratio past `--buffer-size` (0.0-1.0). | `0.2` |
| `--disk-cache-dir` | | Disk cache directory. | `~/.mountOS/cache` |
| `--disk-cache-size` | | Max disk cache (MiB/GiB/TiB). | freeDisk/10, clamped [10G, 50G] |
| `--disable-cache-dir` | | Disable disk caching. | false |
| `--eviction-policy` | | Disk segment eviction: `2-random` or `lru`. | `2-random` |
| `--max-dirty-bytes` | | Dirty bytes before auto-flush. | buffer-size/4, clamped [64M, 512M] |
| `--meta-open-connections` / `--meta-idle-connections` / `--meta-idle-wait-time` | | Per-dataserv TCP pool tuning. | `2` / `2` / `120` |
| `--attr-cache` / `--entry-cache` / `--dir-entry-cache` | | Kernel metadata cache TTLs (seconds). Defaulted knobs adapt automatically (2x on read-only mounts; up to 15s while the mount is the volume's exclusive client). | `1.0` / `1.0` / `1.0` |
| `--negative-entry-cache` | | Negative entry cache TTL (0-600 s). | `1` |
| `--umask` | | Octal umask override (for example `022`). | |
| `--uid` / `--gid` | | Owner uid/gid when mountos runs as root (defaults to `SUDO_UID`/`SUDO_GID` or process id). | `-1` (unset) |
| `--agent` | | pprof debug agent (port 6060-6160). | true |
| `--fork-name` | | Fork to mount (empty = main). | |
| `--temporary-fork[=NAME]` | | Ephemeral per-session fork (`auto` generates one). | off |
| `--session-audit` / `--session-audit-interval` | `SESSION_AUDIT...` | Session activity audit logging and interval (`5m`, `10m`, `15m`, `30m`, `1h`, `3h`, `6h`, `12h`, `1d`). | false / `15m` |

**Backend selection** (mutually exclusive): macOS `--macfuse`, `--fskit, -k`, `--nfs, -N`, `--smb`; Linux `--nfs, -N`, `--io-uring`. Related tuning includes `--nfs-port`, `--nfs-nconnect`, `--smb-port`, `--io-uring-queue-depth` (64), `--io-uring-buffer-count` (32), `--io-uring-max-queues` (0=all CPUs), `--io-uring-sqpoll`.

**POSIX/permissions**: `--acl` (implies `--xattr`), `--xattr`, `--ioctl`, `--xattr-security` (comma-separated sensitive namespaces, for example `selinux,trusted` on Linux, `apple` on macOS), `--null-permissions`, `--groups-cache` (300s).

**Embedded gateway** (per-volume S3/HDFS for local clients): `--gateway` (`s3,hdfs`), `--gateway-only`, `--gateway-no-loopback` (binds 0.0.0.0, requires `--gateway-cert`/`--gateway-key`), `--gateway-port`, `--gateway-object-meta` (true), `--gateway-object-tags` (true), `--gateway-keep-etag` (true), `--gateway-trace`, `--gateway-allow-reuse-port`. Iceberg REST/S3 listeners are derived automatically from the volume type and bind `127.0.0.1`.

**Windows backend** (`--backend`): `mountosio`, the native kernel filesystem driver.

## Deployment-level (compose) variables

A separate deployment env file carries infrastructure-level settings rather than per-service tuning. Observed keys include:

- PostgreSQL: `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`, plus an auth DB triple `AUTH_POSTGRES_USER`/`AUTH_POSTGRES_PASSWORD`/`AUTH_POSTGRES_DB`.
- DNS/endpoints: `API_URL`, `AUTH_URL`, `ADMIN_URL`, `METRICS_HOST`, `AUTH_ENDPOINT`, `ADMIN_PORT`, `ADMIN_ENDPOINT`, `ADMIN_DISABLE_LOCALHOST`.
- Identity provider and admin console wiring.
- nginx-proxy-manager DB settings (`NPM_DB_NAME`, `NPM_DB_USER`, `NPM_DB_PASSWORD`, `NPM_DB_ROOT_PASSWORD`).
- AWS KMS for envelope encryption (`AWS_KMS_ARN`, `AWS_KMS_REGION`) plus AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).
- `LOG_LEVEL`, `DEFAULT_EMAIL`, `BUILD_CLIENT_ID`.

Redacted `.env-sample` templates are produced from each service's `.env` by a helper that walks every non-comment `KEY=value` line, blanks the value, and preserves keys and comments. This is the supported way to publish a redacted template.

## Configuration precedence (mountos)

For credentials, `mountos` prefers explicit `--access-key-id` / `--secret-access-key` flags (both must be provided together), falling back to `MOUNTOS_ACCESS_KEY_ID` / `MOUNTOS_SECRET_ACCESS_KEY`. The discovery URL comes from `--discovery-url` or `MOUNTOS_DISCOVERY_URL`, both empty is an error.


<!-- topic: database-schema -->

# Database and Schema

mountOS stores all control-plane and metadata state in a SQL database. The
filesystem block/object data lives in object storage, not in the database. This
page covers which engines are supported, how the schema is split, where the
definitions live in the repo, and how schema generation and migration work.

## Supported engines

mountOS maintains two source dialects (MySQL and PostgreSQL) and derives several
compatible variants from them. Two families exist, the MySQL family and the
PostgreSQL family.

| Engine | Family | How it is produced |
|---|---|---|
| PostgreSQL | postgresql | Source of truth |
| MySQL | mysql | Source of truth |
| TiDB | mysql | Uses the MySQL dialect (wire-compatible), reuses the MySQL schema and query code directly |
| MariaDB | mysql | Generated from MySQL via hint directives |
| Vitess | mysql | Generated from MySQL via hint directives |
| CockroachDB | postgresql | Generated from PostgreSQL via hint directives |
| YugabyteDB | postgresql | Generated from PostgreSQL via hint directives |

Six engines have their own generated schema trees (`mysql`, `mariadb`, `vitess`
from the MySQL source; `postgresql`, `cockroachdb`, `yugabytedb` from the
PostgreSQL source). TiDB does not get a separate tree, it runs against the
MySQL schema and query code. At runtime, engine-specific behavior is keyed by a
SQL provider value, and the provider set recognized by the capability layer is
broader than the generated schema trees. Each provider declares which
capabilities it supports and from which minimum version, so query routing adapts
per engine.

### Recognized SQL providers

The capability layer recognizes exactly eleven provider values, grouped by the
two wire dialects. Each provider maps to one dialect (its wire protocol), and a
provider whose declared dialect does not match the configured dialect is
rejected at startup.

| Provider value | Dialect | Accepted config aliases |
|---|---|---|
| `mysql` | mysql | `mysql` |
| `mariadb` | mysql | `mariadb` |
| `tidb` | mysql | `tidb` |
| `singlestore` | mysql | `singlestore`, `memsql` |
| `aurora-mysql` | mysql | `aurora-mysql`, `aurora_mysql`, `auroramysql` |
| `vitess` | mysql | `vitess` |
| `postgresql` | postgresql | `postgresql` |
| `cockroachdb` | postgresql | `cockroachdb`, `cockroach`, `crdb` |
| `yugabytedb` | postgresql | `yugabytedb`, `yugabyte`, `yb` |
| `alloydb` | postgresql | `alloydb`, `alloy` |
| `aurora-postgresql` | postgresql | `aurora-postgresql`, `aurora_postgresql`, `aurorapostgresql`, `aurora-postgres` |

The providers beyond the engines with generated schema trees (SingleStore,
Aurora MySQL, AlloyDB, Aurora PostgreSQL) do not get their own generated schema
tree. They run against the schema of their base dialect (MySQL or PostgreSQL)
and are distinguished only by their capability profile. An unrecognized provider
value is a hard error (capability init panics, and a direct capability check
returns false) rather than being silently treated as a base engine.

### Capability and version routing

Each (provider, capability) cell in the runtime matrix is one of:

- always supported (version ignored),
- never supported (version ignored, routed to the fallback variant),
- version-gated by a minimum `major.minor.patch`.

For a version-gated cell, the routing depends on whether a provider version is
configured/detected:

- If a version is known, the capability is enabled only when the running
  version is at least the declared minimum.
- If the version is unknown (unset or unparseable), the cell falls back to a
  per-cell default (`defaultWhenUnknown`). That default is `true` in only three
  places in the matrix (PostgreSQL `SKIP LOCKED`, MySQL `JOIN LATERAL`, and
  Aurora MySQL `JOIN LATERAL`). For every other version-gated cell the unknown
  default is `false`, so the engine is routed to the fallback query variant
  until the real version is read.

The actual version is queried from the live database (`SELECT VERSION()` on the
MySQL family, `SHOW server_version` on the PostgreSQL family), normalized to a
bare `major.minor.patch` (vendor suffixes like `-TiDB-v8.1.0` or
`(Debian ...)` are stripped), and the capability set is then recomputed.

Representative capability rules (exact minimums from the runtime matrix):

- Window functions: not supported on Vitess; version-gated on MySQL 8.0+,
  MariaDB 10.2+, and Aurora MySQL 8.0+; always supported on TiDB, SingleStore,
  and the entire PostgreSQL family.
- `SELECT ... FOR UPDATE SKIP LOCKED`: MySQL 8.0.1+, Aurora MySQL 8.0.1+,
  MariaDB 10.6+, PostgreSQL 9.5+ (unknown-version default true), CockroachDB
  22.1+, YugabyteDB 2.13+; always supported on AlloyDB and Aurora PostgreSQL;
  not supported on TiDB, SingleStore, or Vitess.
- `FOR UPDATE OF`: version-gated on MySQL 8.0+ and Aurora MySQL 8.0+, always
  supported on the rest of the PostgreSQL family (PostgreSQL, CockroachDB,
  YugabyteDB, AlloyDB, Aurora PostgreSQL); not used on MariaDB, SingleStore,
  Vitess, or TiDB (TiDB requires a table name rather than an alias, so it is
  routed to the fallback to avoid a third query variant).
- `JOIN LATERAL`: always supported on the PostgreSQL family; version-gated on
  MySQL 8.0.14+ and Aurora MySQL 8.0.14+ (both default true when the version is
  unknown); never used on MariaDB, TiDB, SingleStore, or Vitess (those advertise
  window functions but reject `JOIN LATERAL` syntactically).
- Pipelined queries (multi-query in one round trip): always supported on the
  PostgreSQL family (via the pgx batch interface); never on the MySQL family.
- Multi-statement execution and single-statement meta operations: never
  supported across all providers in the current matrix (the PostgreSQL family
  uses batching instead, and the MySQL driver path runs statements
  sequentially).

When a capability is absent, the engine is routed to a fallback query variant
rather than failing.

## Admin vs data schema split

The schema is divided by responsibility. Within a single Postgres deployment,
these map to two database schemas (`mountos_admin` for admin, `mountos_data` for
data/region).

| Layer | Purpose | Representative tables |
|---|---|---|
| admin | Global control plane (accounts, users, regions, service discovery, licensing, audit) | `accounts`, `users`, `regions`, `admin_storages`, `account_volumes`, `service_nodes`, `admin_audit_logs`, `admin_outbox`, `region_clusters`, `license_quota_state`, `licenses`, `hub_nodes`, `client_sessions`, `service_alerts` |
| data (metadata) | Per-volume filesystem metadata, sharded and partitioned by `sid` (shard id) | `nkeys`, `nodes`, `attrs`, `blobs`, `segment_counters`, `xattrs`, `acls`, `forks`, `fork_view`, `sessions`, `plocks`, `flocks`, `event_log` |
| data (region) | Region-scoped operational state (storage bindings, volume keys, sharing, multipart, GC queues) | `storages`, `volumes`, `volume_api_keys`, `shared_links`, `multipart_uploads`, `multipart_parts`, `audit_logs`, GC queue tables, `region_outbox`, `resource_leases`, `region_node_heartbeats` |
| data (lake) | Iceberg REST catalog tables | `lake_namespaces`, `lake_tables`, `lake_snapshots`, `lake_refs`, `lake_views`, `lake_view_versions`, `lake_scan_plans`, `lake_metric_reports`, `lake_keys` |

The metadata, region, and lake tables all live in the data schema tree. The
region tables form their own query package, while the lake tables live in the
data query package alongside the core metadata queries.

The metadata tables are hash-partitioned by `sid` in PostgreSQL. `nodes`,
`attrs`, and `event_log` use 32 partitions; `blobs`, `segment_counters`, and
`xattrs` use 8. This multi-tenant sharding is a PostgreSQL feature. Derived
engines that cannot partition this way (CockroachDB, YugabyteDB) skip the
partitioning via hint directives and use a single table.

## Where the schema definitions live

Two trees exist in `mountos-servers`. Only one is hand-edited.

```
db/schemas/{mysql,postgresql}/{admin,data}/*.sql                                 <- source of truth (edit here)
schemas/{mysql,postgresql,mariadb,vitess,cockroachdb,yugabytedb}/{admin,data}/   <- generated (do not edit)
```

Source PostgreSQL files (the canonical reference):

- `db/schemas/postgresql/admin/001-admin-schema.sql` (which also defines the
  `licenses` table) and numbered siblings (`002-client-sessions.sql`,
  `003-hub-nodes.sql`, `004-service-alerts.sql`, `005-client-sessions-search.sql`,
  `006-volume-size-history.sql`, `007-license-quota.sql`, `008-region-clusters.sql`)
- `db/schemas/postgresql/data/001-metadata-schema.sql` (filesystem metadata)
- `db/schemas/postgresql/data/002-region-schema.sql` (region operational state)
- `db/schemas/postgresql/data/003-lake.sql` (Iceberg catalog)

The MySQL source mirror lives under `db/schemas/mysql/{admin,data}/`. Each source
file lists its tables in a header comment, which is the fastest way to scan the
schema without reading full DDL.

The generated per-engine output under the top-level `schemas/` directory is
concatenated and dialect-adjusted. It must not be edited directly because it is
overwritten on every regeneration.

## Query and codegen layout (sqlc)

SQL queries are stored separately from schemas and compiled to typed Go with
sqlc. Configuration is `db/sqlc.yaml`.

```
db/queries/{mysql,postgresql}/{admin,data,region}/*.sql   <- query definitions
internal/db/sqlc/{mysql,postgresql}/{admin,data,region}/  <- generated Go
```

The sqlc config generates both MySQL and PostgreSQL output for every layer
(admin, data, region) on each run. Generating both unconditionally is intentional
so that a query that breaks one dialect is caught at build time. Lake queries
live inside the `data` query package (`db/queries/{mysql,postgresql}/data/lake_*.sql`),
not a separate tree. Both engines target the `database/sql` interface in the
generated Go. At runtime PostgreSQL uses the `pgx/v5/stdlib` driver (registered
as `pgx`, default port 5432) and MySQL (also serving TiDB) uses
`github.com/go-sql-driver/mysql` (default port 3306). Type overrides in the
config map binary and JSONB columns to Go types compatible with these drivers.

Query linting uses sqlc vet with CEL-based rules in `db/sqlc-vet.yaml` (sqlc's
prepare-based check does not work with the `sqlc.arg()` macros this codebase
uses).

### Dialect abstraction layer

On top of the per-dialect sqlc output, a generated dialect abstraction layer in
`internal/db/dialect` exposes one unified query interface that dispatches to the
MySQL or PostgreSQL implementation at runtime. The active engine is selected from
service configuration (a SQL dialect plus a provider and optional provider
version), so callers use the same Go interface regardless of engine. TiDB callers
select the MySQL dialect. PostgreSQL emits `RETURNING` and `$N` placeholders;
MySQL emits `?` placeholders and avoids `RETURNING`. Capability checks (for
example window-function, `SKIP LOCKED`, or `JOIN LATERAL` support) drive which
query variant is used per provider, using the version-routing rules described
above. This layer and its sub-packages are generated, not hand-edited.

## Generation and migration flow

Schema generation is driven from the source PostgreSQL and MySQL trees using
inline hint directives, then sqlc generates Go from the queries.

```
db/schemas/{mysql,postgresql}/*/*.sql   (source, with hint directives)
        | make generate-schema  (scripts/schema/gen.go)
schemas/{mysql,postgresql}/*/schema.sql (concatenated)
        |-- hints applied --> schemas/{mariadb,vitess}/   (from mysql source)
        |-- hints applied --> schemas/yugabytedb/         (from postgresql source)
        +-- hints applied --> schemas/cockroachdb/        (from postgresql source)
```

The MySQL source produces the `mysql`, `mariadb`, and `vitess` trees; the
PostgreSQL source produces the `postgresql`, `cockroachdb`, and `yugabytedb`
trees.

Relevant Make targets:

| Target | Action |
|---|---|
| `make generate-schema` | Generate combined and per-engine schema files from source |
| `make sqlc-generate` | Run `generate-schema`, then sqlc codegen plus the dialect abstraction layer |
| `make sqlc-vet` | Lint queries with sqlc vet (CEL rules) |
| `make db-start` / `make db-stop` | Start / stop validation database containers |
| `make db-validate` | Start validation databases, run SQL query validation, write a report, stop databases |
| `make db-validate-ci` | Same validation without log tee (CI mode, fails fast) |
| `make db-clean` | Stop databases and remove reports and validation logs |

### Hint directives

Source SQL files carry inline directives that conditionally include or exclude
blocks per target engine, so engine-specific incompatibilities (hash
partitioning, PL/pgSQL `DO` blocks, type-creation syntax) are handled in one
place instead of post-generation patches.

| Directive | Effect |
|---|---|
| `--@begin-skip <targets>` / `--@end-skip` | Skip the enclosed lines for the listed targets |
| `--@begin-only <targets>` / `--@end-only` | Include the enclosed lines only for the listed targets |

Targets are a comma- or space-separated list of engine names (for example
`cockroachdb`, `yugabytedb`, `mariadb`, `vitess`, and also the base names
`postgresql`/`mysql`). A directive applies only to the engines it explicitly
names, so a base dialect is affected only when its own name appears in the list.
All `--@` lines are stripped from the generated output.

For example, CockroachDB and YugabyteDB skip `PARTITION BY HASH (sid)` and the
PL/pgSQL partition-creation loops, and replace guarded `CREATE TYPE` blocks with
engine-native forms.

### Migration model

Schemas use idempotent `CREATE TABLE IF NOT EXISTS` and
`CREATE INDEX IF NOT EXISTS` statements throughout, so applying a schema file is
re-runnable. The schema is designed as a single source of truth with no
`ALTER TABLE` migration chain (new columns are added inline to the table
definition so a fresh setup produces the final shape in one pass). There is no
incrementally versioned migration tool in this tree (no numbered up/down
migration deltas). The numeric file prefixes (`001-`, `002-`) order
concatenation, not a migration history.


<!-- topic: secrets -->

# Vaults and Secrets

mountOS services never embed credentials. Every service reads its secrets at
startup from an external secret store through a single provider-agnostic
interface, so an operator supplies their own keys, database URLs,
and storage credentials into a vault of their choice. The vault is required, not
optional. Every service-to-service handshake reads from it.

## The SecretStore abstraction

All services talk to a `SecretStore` interface that hides backend differences
behind a logical path scheme. The active backend is selected by the
`VAULT_PROVIDER` environment variable. A missing or unknown value causes a fatal
exit at startup.

| Backend | `VAULT_PROVIDER` | Required env vars | Optional (cross-cloud) |
| --- | --- | --- | --- |
| HashiCorp Vault | `hashicorp` | `VAULT_HASHICORP_ADDRESS`, `VAULT_HASHICORP_ROLE_ID`, `VAULT_HASHICORP_SECRET_ID` | - |
| AWS Secrets Manager | `aws` | `VAULT_AWS_REGION` | `VAULT_AWS_ACCESS_KEY_ID`, `VAULT_AWS_SECRET_ACCESS_KEY` |
| GCP Secret Manager | `gcp` | `VAULT_GCP_PROJECT_ID` | `VAULT_GCP_CREDENTIALS_FILE` |
| Azure Key Vault | `azure` | `VAULT_AZURE_URL` | `VAULT_AZURE_TENANT_ID`, `VAULT_AZURE_CLIENT_ID`, `VAULT_AZURE_CLIENT_SECRET` |

For AWS, GCP, and Azure, mountOS uses the cloud SDK's standard credential chain
by default (instance role, Workload Identity, managed identity, and so on). The
optional explicit credential variables exist only for cross-cloud deployments
(for example running on GCP but reading secrets from AWS Secrets Manager).
Providers can be mixed across regions, and each vault is configured
independently.

Backend-specific behavior:

- **HashiCorp Vault** uses a KVv2 secrets engine mounted at `mountos/` and
  AppRole auth (`auth/approle/login`), with automatic token renewal at roughly
  75% of the token TTL (falling back to a 5-minute cadence when the TTL cannot be
  looked up). Reconnection on auth errors (HTTP 401/403) is deduplicated via
  singleflight to avoid a thundering herd.
- **AWS Secrets Manager** maps logical paths directly to secret names (prefixed
  with `mountos/`, with `/` preserved) and uses the `AWSCURRENT` / `AWSPREVIOUS`
  staging labels for versioning.
- **GCP Secret Manager** secret IDs cannot contain `/`, so the logical path is
  encoded with double-underscore (`__`) separators under a `mountos__` prefix
  (`mountos/dataserv` becomes `mountos__dataserv`, `mountos/volcreds/<vol>`
  becomes `mountos__volcreds__<vol>`). Versions use GCP integer version numbers,
  read newest-first among enabled versions.
- **Azure Key Vault** secret names must match `[A-Za-z0-9-]+`, so the path is
  encoded with double-dash (`--`) separators under a `mountos--` prefix, and
  versions are ordered by created date. Deletion is soft-delete plus purge.
  Because per-secret ACLs are not native to access policies, mountOS splits each
  path group into its own vault (`mountos-vault-services`,
  `mountos-vault-verifiers`, `mountos-vault-s3creds`, `mountos-vault-volcreds`,
  `mountos-vault-api-master`), and `VAULT_AZURE_URL` points at the vault for that
  service's path group.

Because path segments must survive all four backends, secret sub-paths use
`[a-z0-9-]` only. Underscores collide with the Azure naming rules (Azure rewrites
`_` to `-`).

### Resource namespace prefix

Every name shown above is rooted on the literal `mountos`. The optional
`VAULT_RESOURCE_PREFIX` environment variable (1-11 lowercase alphanumeric/hyphen
characters, empty by default) namespaces that root to `mountos-<prefix>`, so
multiple mountOS deployments can share one cloud account, project, or
subscription without their secret names colliding. With `VAULT_RESOURCE_PREFIX`
unset, naming is byte-identical to the table above. With it set to `acme`:
HashiCorp mounts the KVv2 engine at `mountos-acme/`, AWS Secrets Manager names
become `mountos-acme/<path>`, GCP Secret Manager IDs become
`mountos-acme__<path>`, and Azure Key Vault names become `mountos-acme--<path>`
(and the split-vault names become `mountos-acme-vault-services`, and so on).
This value must be set identically on every service in a deployment (they
resolve each other's secrets by name) and must match the `resource_prefix`
Terraform variable used to provision that deployment's cloud IAM policies,
which scope access by the same prefixed name pattern.

### Version identity across backends

The four backends do not expose a uniform monotonic ordinal for a secret's
versions (HashiCorp KVv2 returns an integer, AWS Secrets Manager uses opaque
UUIDs, GCP returns its own integer, Azure uses string ids). mountOS normalizes
this by treating each version's creation timestamp as the single cross-backend
version identity. Every read returns the version's `CreatedTime`, and any
integer version used elsewhere in mountOS is the Unix-seconds value of that
timestamp. This normalization is uniform across every read path and every
consumer (see the master-key rotation section for how it is applied end to end).

## Kinds of secrets

| Secret | Lives in | Created by | Purpose |
| --- | --- | --- | --- |
| Service own secrets (`<service>`) | Hub + Region | operator | DB connection settings, service ED25519 keypair |
| `service-verifiers` | Hub (source of truth), replicated to Region | operator | Public keys for cross-service JWT verification |
| `api-master` | Region (independent per region) | operator | Master string that derives the key encrypting stored API-key ciphertext |
| `s3creds` | Region | dataserv / gcserv | Object-storage backend credentials (for example AWS, Backblaze, MinIO) |
| `volcreds` | Region | dataserv / gcserv | Per-volume encryption keys, write-once |

Service own secrets carry the database settings (`DB_DIALECT`, `DB_URL`,
`DB_PROVIDER`, optional `DB_PROVIDER_VERSION`) and the service's signing and
verification keys. appserv additionally holds `PROVIDER_VERIFICATION_KEY` (the
admin ED25519 public key, used to verify Admin SDK JWTs) and
`DASHBOARD_USER_HMAC_KEY` (a dedicated random secret, at least 32 bytes, that
signs the `X-MountOS-Dashboard-User` header). The dashboard HMAC key is a
distinct secret from `PROVIDER_VERIFICATION_KEY`: the verification key is public,
so an HMAC keyed on it would prove nothing beyond the JWT itself. Share it only
with the trusted dashboard backend. Data-plane services (appserv, dataserv,
gcserv) hold a DB URL; blockserv holds only its keypair. The `mountos` client has
no vault entry at all, it authenticates with an appserv-issued access key pair.

Secrets loaded from the vault at startup are merged over process environment
variables, so a value present in the vault takes precedence over the same
variable in the environment.

### ED25519 key format

ED25519 keys are stored as raw bytes, base64-encoded with standard (not URL-safe)
base64. Wrapped formats (SPKI, DER, PEM, PKCS8) are rejected because decoding
yields the wrong byte length.

| Key type | Raw size | Base64 length | Field |
| --- | --- | --- | --- |
| Signing (private) | 64 bytes (seed + public) | 88 chars | `ED25519_SIGNING_KEY` |
| Verification (public) | 32 bytes | 44 chars | `ED25519_VERIFICATION_KEY` |
| admin verification | 32 bytes | 44 chars | `PROVIDER_VERIFICATION_KEY` |

To extract the raw 32-byte public key from an SPKI DER (base64) value:

```bash
echo "$SPKI_B64" | base64 -d | tail -c 32 | base64
```

## Vault topology

A deployment uses two logical vaults. A **Hub vault** next to the HUB (the
appserv control plane) and one **Region vault** per region. They are independent.
Nothing in mountOS writes across them.

- **Hub vault** holds appserv's own secrets and the source-of-truth
  `service-verifiers` set (one public key per service: appserv, dataserv, gcserv,
  blockserv).
- **Region vault** (one per region, alongside the Meta server cluster) holds the
  regional services' own secrets (dataserv, gcserv, blockserv), a replicated copy
  of `service-verifiers`, a region-independent `api-master`, plus `s3creds` and
  `volcreds`. Region vaults are independent; secrets are not shared across
  regions.

Every service-to-service call carries an ED25519-signed JWT. The receiver
verifies it using the sender's public key from its local `service-verifiers`
path, with the last two versions cached so rotation is zero-downtime. Regional
services never have to call the HUB to verify a JWT, because the replicated
verifier set in their own Region vault is enough.

### Replication

mountOS does not implement vault-to-vault sync. **Replication is the operator's
responsibility.** The `service-verifiers` set must be replicated from the Hub
vault to every Region vault so cross-service JWT verification works across
regions. Use whatever the chosen backend offers (HashiCorp performance or DR
replication, cross-region secret sync, or a scheduled script). The `s3creds` and
`volcreds` entries are written directly into the Region vault by application
code, so they need no replication.

## Permission model

Each service authenticates with its own identity (AppRole, IAM role, service
account, or service principal) and receives only the access it needs.

| Service | Own secrets | service-verifiers | api-master | s3creds | volcreds |
| --- | --- | --- | --- | --- | --- |
| appserv (Hub) | R | R/W | none | none | none |
| dataserv | R | R | R | CRUD | CRD (create + read + delete) |
| gcserv | R | R | R | CRUD | CRD (create + read + delete) |
| blockserv | R | R | R | R | none |

Notes:

- The operator writes new `api-master` versions out of band; all services read
  it only.
- `volcreds` are write-once at the application level. dataserv and gcserv refuse
  to overwrite an existing volume encryption key once the volume holds data,
  because changing the key would require re-encrypting all volume data. A
  volume's key may be operator-supplied (minimum 8 characters) or auto-generated.
- A volume's `volcreds` is deleted only when the volume itself is being removed.
  dataserv deletes it on volume hard-delete, and gcserv reclaims it for a
  deactivated volume after vault-cleanup is enabled, the grace period has passed,
  and meta and storage cleanup have already completed. The deletion is gated on
  the volume actually using encryption.

The HashiCorp policy set is the reference implementation; equivalent IAM policies
(AWS), IAM bindings (GCP), and access policies or RBAC (Azure) express the same
matrix. On Azure, where per-secret ACLs are limited, the recommended layout is
separate vaults per concern (services, verifiers, s3creds, volcreds,
api-master).

## How an operator supplies their own secrets

1. Choose a backend and set `VAULT_PROVIDER` plus the required connection
   variables from the table above.
2. Enable the storage location. For HashiCorp this is a KVv2 engine mounted at
   `mountos/`; for the cloud backends it is the corresponding secret manager
   namespace.
3. Generate one ED25519 keypair per service and write each service's own secret
   (DB settings + signing/verification keys). Write each service's public key
   into `service-verifiers` in the Hub vault and replicate to Region vaults. Seed
   `PROVIDER_VERIFICATION_KEY` into appserv, and a dedicated random
   `DASHBOARD_USER_HMAC_KEY` (at least 32 bytes) into appserv and the dashboard
   backend that signs `X-MountOS-Dashboard-User`.
4. Provision `api-master` in each Region vault (see below).
5. Apply per-service access policies matching the permission matrix.

The `s3creds` and `volcreds` entries are populated automatically by dataserv and
gcserv as volumes and storage backends are configured; the operator only grants
access.

### Provisioning the API master key

`api-master` is a single secret with exactly one field, `key` (a random string of
at least 12 characters, folded into a 32-byte master key via HKDF-Extract with
SHA-256 when read). Adding a second field would corrupt rotation, since every
backend versions the whole blob per write. Per backend:

```bash
# HashiCorp
vault kv put mountos/api-master 'key=<value>'

# AWS
aws secretsmanager put-secret-value --secret-id mountos/api-master --secret-string '{"key":"<value>"}'

# GCP
printf '{"key":"<value>"}' | gcloud secrets versions add mountos__api-master --data-file=-

# Azure
az keyvault secret set --vault-name <vault> --name mountos--api-master --value '{"key":"<value>"}'
```

If `VAULT_RESOURCE_PREFIX` is set for this deployment, replace `mountos` in
every command above with `mountos-<prefix>` (see "Resource namespace prefix").

## Rotation

| Secret | Frequency | Owner | Downtime |
| --- | --- | --- | --- |
| Service ED25519 keys | Weekly or monthly, policy-driven | operator | None (last two keys cached) |
| api-master | On demand, per region, not automatic | operator writes; gcserv re-aligns ciphertext | None (latest versions cached) |
| volcreds | Never (immutable after first write) | - | - |
| s3creds / DB credentials | Per your own policy | operator | Per backend |

Service key rotation: generate new keypairs, update each service's own secret and
the matching `service-verifiers` entry, then replicate verifiers to all regions.
Services keep the last two keys cached, so the old key stays valid through the
next cycle and no restart is required.

API master rotation: the operator writes a new `api-master` version using the
backend's native mechanism or the Admin SDK vault operation. The user-facing API
key string is unchanged; only the stored ciphertext is re-encrypted, and services
keep the latest two versions for a decryption grace window. gcserv runs a
master-key alignment goal that picks up the new version and re-encrypts stored
API-key ciphertexts to it.

Version identity is uniform end to end. A version is identified by the Unix
timestamp derived from its creation time (see "Version identity across
backends"), and that same int64 flows through every step. When dataserv encrypts
a new API-key secret it stamps the row with the current master-key version.
gcserv's alignment goal reads the latest two versions, re-encrypts each stored
ciphertext to the newest one, and persists the new version, with the update
optimistically locked on `(api_key, salt, key_version)` so concurrent writers
cannot clobber each other. Decryption looks up the per-row version among the two
cached versions, which is why the rolling window of the latest two is what makes
rotation downtime-free.

### Admin SDK vault operation

The public Admin SDK exposes a single vault operation, a resync. In the Go
client it is `VaultService.Resync`, mapped to `POST /api/v1/vault/resync`, taking
no request body and returning no payload (success or error only). A matching
TypeScript client exists (`client.vault.resync()`). The SDK is generated from a
published spec at version `1` under base path `/api/v1` (Go module
`github.com/mountos-io/mountos-admin-sdk/go`). Authentication is the operator's
ED25519-signed JWT, and the endpoint additionally requires system-admin scope.

Resync: after a rotation the operator can refresh vault-backed caches (service
verifier keys plus the master key cache) via that endpoint. It is rate-limited to
2 per minute. The handler refreshes appserv's local verifier cache and then
broadcasts to the other services to refresh theirs. Epoch-based deduplication
prevents redundant refreshes from broadcast fan-out.


<!-- topic: download-install -->

# Download and Install

mountOS ships as prebuilt binaries served from a public distribution endpoint. A client CLI (`mountos`) covers macOS, Linux, and Windows. Server services (`*serv`) ship as Linux binaries. One-line installer scripts handle download, checksum verification, extraction, and placement.

## Distribution Endpoint

| Item | Value |
| --- | --- |
| Public distribution base URL | `https://download.mountos.sh` |
| bash installer | `https://download.mountos.sh` (also `https://download.mountos.sh/mountos.sh`) |
| PowerShell installer | `https://download.mountos.sh/install.ps1` |

`download.mountos.sh` is a Cloudflare R2 bucket (the `download` subdomain of `mountos.sh`) that serves both the installer scripts and the release artifacts. `https://download.mountos.sh/` and `https://download.mountos.sh/mountos.sh` return the bash installer; `https://download.mountos.sh/install.ps1` returns the PowerShell installer. Release archives, checksums, and manifests live under `/dist`.

## Quick Install

macOS / Linux (bash):

```sh
curl -fsSL https://download.mountos.sh | bash
```

Windows (PowerShell):

```powershell
Invoke-WebRequest https://download.mountos.sh/install.ps1 -OutFile mountos.ps1; .\mountos.ps1
```

By default this installs the full client bundle (`mountos-macos`/`mountos-windows`: GUI apps plus the `mountos` CLI) on macOS and Windows, and the bare `mountos` CLI on Linux (no bundled GUI suite exists there), latest version, auto-detecting platform and architecture. Pass `--pkg mountos-cli` (`-Pkg mountos-cli`) to install just the CLI on macOS/Windows too.

## Installer Options

The bash installer accepts options after `-- `:

```sh
curl -fsSL https://download.mountos.sh | bash -s -- --pkg mountos-cli --version 1.2.3
```

| bash flag | PowerShell flag | Meaning | Default |
| --- | --- | --- | --- |
| `--pkg NAME` | `-Pkg NAME` | Package to install | full client bundle (`mountos-macos`/`mountos-windows`), `mountos-cli` on Linux |
| `--version X.Y.Z` | `-Version X.Y.Z` | Install a specific version | latest |
| `--major N` | `-Major N` | Latest of major series N | (unset) |
| `--platform NAME` | (auto) | Override platform (`linux`, `darwin`, `windows`) | detected |
| `--arch NAME` | `-Arch NAME` | Override arch (`amd64`, `arm64`) | detected |
| `--prefix PATH` | `-Prefix PATH` | Install directory | see below |
| `--list` | `-List` | List available versions for the package | n/a |

The bash installer validates `--pkg`, `--version`, `--platform`, and `--arch` values against `^[A-Za-z0-9._-]+$`, and `--major` must be numeric.

Default install prefix:

- bash: `/usr/local/bin` (if the prefix exists but is not writable the installer aborts and suggests `sudo` or `--prefix $HOME/.local/bin`)
- PowerShell: `%LOCALAPPDATA%\mountOS\bin` (added to the user `PATH` automatically). Before installing, the PowerShell installer performs a write test on an existing prefix and aborts with an Administrator / `-Prefix` hint if it is not writable.

List available versions:

```sh
curl -fsSL https://download.mountos.sh | bash -s -- --list
```

The bash installer uses `jq` when present for version sorting and manifest parsing, and falls back to `grep`/`sed` otherwise. It requires `sha256sum` or `shasum` for integrity verification.

## Platform and Architecture Matrix

Platform and arch are auto-detected (`uname` on bash, `PROCESSOR_ARCHITECTURE` on PowerShell) and can be overridden.

| Platform | Arch values | Archive format | Notes |
| --- | --- | --- | --- |
| `darwin` | `arm64`, `amd64` | `.tar.gz` (`mountos-cli`), `.pkg` (`mountos-macos`) | `mountos-cli` is per-arch (matches Linux/Windows); `mountos-macos` is one universal pkg, its bundled CLI is lipo'd |
| `linux` | `amd64`, `arm64` | `.tar.gz` | |
| `windows` | `amd64`, `arm64` | `.zip` (`mountos-cli`), `.exe` (`mountos-windows`) | requires `unzip` (bash) or `Expand-Archive` (PowerShell) for the CLI zip |

The installer verifies the downloaded archive against a published SHA-256 (carried in the manifest, or a sibling `.sha256` file) before extracting. If no SHA-256 is available the installer warns and skips the integrity check rather than failing.

## Packages

| Package (`--pkg`) | Contents | Platforms |
| --- | --- | --- |
| `mountos-cli` | `mountos` client CLI binary | darwin-arm64/amd64, windows-amd64/arm64 |
| `mountos-macos` | Signed, notarized `.pkg`: `mountOSFSKit.app` (FSKit, auto-deselected below macOS 26) and `mountOS Desktop.app` are each independently selectable at customize-at-install; the `mountos` CLI is shown but mandatory (checked, not uncheckable). Opens directly via Installer.app, no unpack/`install.sh` step. | darwin-universal (one pkg for arm64 and amd64) |
| `mountos-windows` | Signed installer `.exe` (WiX Burn bundle): chains the `mountosio` kernel driver installer, the `mountos` CLI, and the license-acceptance screen | windows-amd64/arm64 |
| `mountos-dataserv` | dataserv server binary | linux-amd64/arm64 |
| `mountos-gcserv` | gcserv server binary | linux-amd64/arm64 |
| `mountos-blockserv` | blockserv server binary | linux-amd64/arm64 |
| `mountos-appserv` | appserv server binary | linux-amd64/arm64 |

The Kubernetes CSI node driver is not a separate package. It is `mountos kubernetes` inside the client binary, and its container image is built from `deploy/csi/Dockerfile`.

Server services ship as Linux binaries only (both architectures). The standalone client (`mountos-cli`) ships on every platform, per-arch. `mountos-macos` is one universal pkg (its bundled CLI is lipo'd from both arch slices); `mountos-windows` is still published per-arch (x64/arm64), Windows has no fat-binary equivalent. Every package archive bundles a license file alongside the binaries (the `.pkg`/installer show it as an accept/decline screen; other archives carry it as a plain file).

### Versioning

`mountos-servers/VERSION` is the canonical version for Go-binary artifacts only. That covers
`mountos-cli` and every `*serv` package, across all platforms. It also names the outer `mountos-macos`/
`mountos-windows` installer/bundle version. It does **not** govern the version of what's bundled
*inside* those installers: `mountOSFSKit.app` (Xcode `MARKETING_VERSION`, bumped independently via
`mountos-osx`'s `make bump-fskit`), `mountOS Desktop.app` (its own `tauri.conf.json` version), and
the Windows driver/installer (its own INF/installer version, changed rarely). Each component keeps
its own release lifecycle.

### Java SDK (Maven)

The HDFS-compatible SDK (`io.mountos:hadoop-mountos`) is not distributed via the installer
scripts above. It publishes to a standard Maven repository layout on the same distribution
bucket (`<base>/maven/`), with the usual `maven-metadata.xml` for version resolution:

```xml
<repository>
  <id>mountos</id>
  <url>https://download.mountos.sh/maven</url>
</repository>
<dependency>
  <groupId>io.mountos</groupId>
  <artifactId>hadoop-mountos</artifactId>
  <version>1.2.3</version>
  <classifier>hadoop3</classifier> <!-- pick hadoop3 or hadoop2; the jars are classifier artifacts -->
</dependency>
```

Install a server binary with the same installer:

```sh
curl -fsSL https://download.mountos.sh | bash -s -- --pkg mountos-dataserv --version 1.2.3
```

## Artifact Layout

Artifacts and manifests follow a stable key layout under the distribution base. Given `<base> = https://download.mountos.sh`:

| Path | Purpose |
| --- | --- |
| `<base>/dist/<pkg>/<version>/<platform>-<arch>.<ext>` | The release archive (`tar.gz` or `zip`) |
| `<base>/dist/<pkg>/<version>/<platform>-<arch>.sha256` | SHA-256 checksum for that archive (`<hex>  <platform>-<arch>.<ext>`) |
| `<base>/dist/<pkg>/latest.json` | Latest-version manifest (all platforms) |
| `<base>/dist/<pkg>/v<major>/latest.json` | Latest within a major series |
| `<base>/dist/<pkg>/versions.json` | Full version list plus `latest` |

Example archive URL:

```
https://download.mountos.sh/dist/mountos-cli/1.2.3/darwin-arm64.tar.gz
```

`latest.json` manifest shape:

```json
{
  "version": "1.2.3",
  "released_at": "2026-01-01T00:00:00Z",
  "artifacts": {
    "linux-amd64": {
      "url": "https://download.mountos.sh/dist/mountos-cli/1.2.3/linux-amd64.tar.gz",
      "sha256": "<hex>",
      "size": 12345678
    }
  }
}
```

`versions.json` shape:

```json
{ "latest": "1.2.3", "versions": ["1.0.0", "1.1.0", "1.2.3"] }
```

The installer resolves `latest.json` (or the pinned `--version` path) for the detected `platform-arch`, downloads the artifact, checks the SHA-256, then extracts. If the archive contains an `install.sh` (macOS), an `install.ps1`, or a `.msi` (Windows), that runs; otherwise executable files are copied into the install prefix. The PowerShell installer runs a bundled `.msi` via `msiexec /i ... /qb` with an elevation prompt, and after a plain binary drop it appends the prefix to the user `PATH` and prints a hint to restart the terminal (or run `$env:PATH += ';<prefix>'`). The bash installer scans the extracted tree (`find -maxdepth 2`) for executable files and installs each with `install -m755`.

`mountos-macos` (`.pkg`) and `mountos-windows` (`.exe` bundle) are a different shape from the plain-binary packages above: they're not extracted and scanned, they're saved to `~/Downloads` (or the current directory) and handed to the platform's own installer, `open` on macOS (launches Installer.app) or running the `.exe` directly on Windows (elevated), rather than a silent/non-interactive install. Both show a license accept/decline screen before proceeding; `mountos-macos` additionally shows a component customize screen (FSKit and Desktop are independently selectable, FSKit unavailable below macOS 26; the CLI is listed but mandatory, its checkbox can't be unchecked).

## Manual Download

You can bypass the installer and fetch an artifact directly, then verify and extract:

```sh
curl -fSL -o mountos.tar.gz \
  https://download.mountos.sh/dist/mountos-cli/1.2.3/darwin-arm64.tar.gz
curl -fsSL https://download.mountos.sh/dist/mountos-cli/1.2.3/darwin-arm64.sha256
shasum -a 256 mountos.tar.gz
tar -xzf mountos.tar.gz
```

Place the extracted `mountos` binary on your `PATH`.

## Building From Source

The release matrix is produced from the `mountos-servers` repo via Make.

| Target | Result |
| --- | --- |
| `make build` | Build all services natively for the host |
| `make <service>` | Build a single service (e.g. `make dataserv`) |
| `make build-prod` | Full release matrix (every `*serv` for linux amd64/arm64, plus the `mountos` client for darwin arm64/amd64 + linux amd64/arm64 + windows amd64/arm64) |
| `CROSS=linux-arm64 make cross-build` | Cross-compile for one target via Zig (`linux-amd64`, `linux-arm64`, `windows-amd64`, `windows-arm64`) |

The client binary is named `mountos`; per-service server binaries are named after the service (`dataserv`, `gcserv`, etc.). `make build` produces all binaries including the client, and `make build-prod` produces the full release matrix. With `make cross-build`, the `*serv` services are only emitted for Linux targets while the `mountos` client cross-compiles for any target. Cross-compiled binaries land under `bin/<cross-target>/`. Container images for individual services can be built locally with `make docker-<service>` (for example `make docker-dataserv`). These local image builds are separate from the binary release flow.



<!-- topic: support -->

# Support and Compatibility

This page describes how a third party obtains mountOS, how it is versioned and released, and what compatibility expectations apply when integrating or upgrading.

## Installation and the distribution channel

mountOS ships prebuilt binaries through a public installer. On Linux and macOS the entry point is a shell script served at `https://download.mountos.sh`. On Windows the entry point is a PowerShell script served at `https://download.mountos.sh/install.ps1`. The installer is the canonical way to obtain a specific package at a specific version.

```bash
# Install the latest version (default package is the full client bundle on
# macOS/Windows, mountos-cli on Linux)
curl -fsSL https://download.mountos.sh | bash

# Install a specific package
curl -fsSL https://download.mountos.sh | bash -s -- --pkg <name>

# Pin an exact version
curl -fsSL https://download.mountos.sh | bash -s -- --pkg <name> --version X.Y.Z

# Pin to a major series, take the latest within it
curl -fsSL https://download.mountos.sh | bash -s -- --pkg <name> --major N

# List available versions for a package and exit
curl -fsSL https://download.mountos.sh | bash -s -- --pkg <name> --list
```

Shell installer flags:

| Flag | Meaning |
|------|---------|
| `--pkg NAME` | Package to install (default: full client bundle, `mountos-macos`/`mountos-windows`; `mountos-cli` on Linux) |
| `--version X.Y.Z` | Install a specific version (default is latest) |
| `--major N` | Take the latest version within major series N |
| `--platform NAME` | Override platform detection (`linux`, `darwin`, `windows`) |
| `--arch NAME` | Override arch detection (`amd64`, `arm64`) |
| `--prefix PATH` | Install prefix (default `/usr/local/bin`) |
| `--list` | List available versions for `--pkg` and exit |

The Windows PowerShell installer takes the same concepts with PowerShell-style flags (`-Pkg`, `-Version`, `-Major`, `-Arch`, `-Prefix`, `-List`); its default install directory is under `%LOCALAPPDATA%\mountOS\bin`.

The installer auto-detects OS (`uname -s`) and architecture (`uname -m`, mapping `x86_64` to `amd64` and `aarch64`/`arm64` to `arm64`), used for every per-arch package. `mountos-macos` is the one exception: it always resolves to `darwin-universal` regardless of detected arch, since its outer `.pkg` bundles a lipo'd universal CLI alongside the already-universal FSKit and Desktop apps. It downloads the matching artifact, verifies a published SHA-256 (either inlined in the manifest or a side-car `.sha256` file) using `sha256sum` or `shasum`, then installs. If neither checksum tool is present the install aborts rather than proceeding unverified. When a checksum tool exists but no checksum is published for the artifact, the installer prints a warning and proceeds. `mountos-macos` (`.pkg`) and `mountos-windows` (`.exe`) are handed directly to the OS installer (saved to `~/Downloads`, then `open` on macOS or run elevated on Windows) instead of being extracted.

Each package publishes machine-readable manifests in the distribution store. Relative to the public distribution base URL:

- `dist/<pkg>/versions.json` lists all known versions plus a `latest` field.
- `dist/<pkg>/latest.json` is the current-version manifest (fields `version`, `released_at`, and an `artifacts` map keyed by `<platform>-<arch>` with `url`, `sha256`, and `size`).
- `dist/<pkg>/v<MAJOR>/latest.json` is the per-major latest manifest.

Integrators that want to detect updates programmatically can read these manifests instead of scraping the installer.

## Package and artifact naming

Each shippable component is published as a separate package. Artifacts live at `dist/<pkg>/<version>/<platform>-<arch>.<ext>` with a matching `.sha256` side-car.

| Package | Contents | Artifact pattern |
|---------|----------|------------------|
| `mountos-cli` | Filesystem client CLI (`mountos`) for Linux, macOS, and Windows | `dist/mountos-cli/<version>/linux-<arch>.tar.gz`, `dist/mountos-cli/<version>/darwin-<arch>.tar.gz`, `dist/mountos-cli/<version>/windows-<arch>.zip` |
| `mountos-macos` | Signed, notarized installer (FSKit, Desktop app, CLI), one universal pkg | `dist/mountos-macos/<version>/darwin-universal.pkg` |
| `mountos-windows` | Signed installer (mountosio kernel driver, CLI) | `dist/mountos-windows/<version>/windows-<arch>.exe` |
| `mountos-<service>` | A server service binary | `dist/mountos-<service>/<version>/linux-<arch>.tar.gz` |

The client CLI ships as the `mountos-cli` package on every platform. The installer's default `--pkg` value is the full client bundle on macOS/Windows (`mountos-macos`/`mountos-windows`) and `mountos-cli` on Linux; pass `--pkg mountos-cli` explicitly for just the CLI on macOS/Windows too.

## Versioning scheme

mountOS uses semantic versioning, `MAJOR.MINOR.PATCH`. The `VERSION` file at the repository root (current value `0.2.0`) is authoritative for Go-binary artifacts only: `mountos-cli` and every `*serv` service package, on every platform, plus the outer `mountos-macos`/`mountos-windows` installer version. Release tooling validates the version against a strict semver pattern (with an optional pre-release suffix) and rejects anything else.

Components bundled inside the macOS/Windows installers keep their own independent version lifecycles, not tied to `VERSION`: the FSKit app (Xcode `MARKETING_VERSION`), the Desktop app (its own `tauri.conf.json` version), and the Windows kernel driver (its own build/installer version fields). The HDFS SDK (`io.mountos:hadoop-mountos`, published to a Maven repository at `<base>/maven` rather than the installer scripts above) is versioned independently too.

| Bump | Example | Use |
|------|---------|-----|
| patch | `0.1.0` to `0.1.1` | backward-compatible fixes |
| minor | `0.1.1` to `0.2.0` | backward-compatible additions |
| major | `0.2.0` to `1.0.0` | breaking changes |

The version is compiled into every service binary and reported at runtime. A `0.x` major series indicates the product is pre-1.0, so breaking changes can land on minor bumps until a `1.0.0` release is cut.

## Platform and architecture support

The release matrix differs by component class.

| Component class | Platforms shipped |
|-----------------|-------------------|
| Server services (`dataserv`, `gcserv`, `blockserv`, `appserv`) | Linux only, `amd64` and `arm64` |
| Filesystem client CLI (package `mountos-cli`) | Linux (`amd64`, `arm64`), macOS (`arm64`, `amd64`), Windows (`amd64`, `arm64`) |
| macOS installer (`mountos-macos`) | macOS universal (single pkg for `arm64` and `amd64`) |
| Windows installer (`mountos-windows`) | Windows `amd64`, `arm64` |

macOS client binaries are code-signed with a Developer ID identity and additionally notarized for Gatekeeper when the release is built with notarization credentials. Windows client binaries are Authenticode-signed with an RFC 3161 timestamp when built with signing credentials.

## Protocol compatibility

Server-to-server traffic uses an internal RPC protocol with an explicit protocol version field. The current protocol version is `2`. The server accepts handshake frames advertising protocol version `0` (unset), `1` (legacy), or `2` (current), and rejects any other value with an "unsupported protocol version" error. This provides a defined backward-compatibility window for rolling upgrades across the legacy and current protocol versions. Clients are first-party mountOS binaries, so this protocol is an internal contract rather than a public integration surface.

## API compatibility surfaces

Third parties integrate through the client's standards-compatible gateway surfaces rather than the internal protocol. Both are started with `mountos gateway --gateway s3,hdfs` (or on a mount, by passing `--gateway` to `mountos mount`).

- S3 gateway (`--gateway s3`): exposes an Amazon S3-compatible API. The authenticator accepts AWS Signature Version 4 (Authorization header, presigned query string, and POST form policy) and AWS Signature Version 2 (header and presigned). A request carrying both header-based and query-based credentials is rejected as ambiguous. Standard SigV4 tooling (AWS CLI, boto3) interoperates. `ListObjects`/`ListObjectsV2` clamp `max-keys` to `1000` (the AWS ceiling); an absent `max-keys` defaults to `1000`, and `max-keys=0` is honored as a bucket probe.
- Multipart upload constraints follow S3 conventions. Every part except the last must be at least 8 MiB (otherwise the server returns `EntityTooSmall`); a part size that is an exact multiple of 8 MiB is recommended for read-aligned assembly but not required. Part numbers and part counts are bounded to `[1, 10000]`. `ListParts` defaults to `1000` parts per page. Clients that cannot meet the part-size rule (for example tools that round non-last part sizes below 8 MiB) cannot complete multipart uploads against the gateway.
- HDFS gateway (`--gateway hdfs`): exposes the WebHDFS REST API for Hadoop-ecosystem clients that speak `WebHdfsFileSystem` over HTTP. The path prefix is `/webhdfs/v1`. The listen port is auto-assigned unless pinned, so pass `--gateway-port hdfs=9870` to match the Hadoop NameNode HTTP convention. Requests are authenticated with AWS Signature Version 4 using service name `hdfs` and a fixed credential-scope region (`auto` by default); clients must sign with the configured region or the request is rejected. Defaults aim to match Hadoop conventions, including a 128 MiB advertised block size and an advertised replication factor of `1` (mountOS handles replication server-side, so this value is informational).

## Upgrade guidance

- Pin by exact version (`--version X.Y.Z`) for reproducible deployments, or pin a major series (`--major N`) to receive compatible updates automatically.
- The installer verifies the published SHA-256 for you. If you fetch an artifact out of band, verify it against the published `.sha256` or the manifest `sha256` field.
- During a fleet upgrade, the protocol version window (accepting versions 0, 1, and 2) lets mixed-version server nodes interoperate while you roll. Plan upgrades so that no node advertises a protocol version outside the accepted set.
- Because the product is in the `0.x` series, treat minor bumps as potentially breaking and read release notes before upgrading across a minor.

## Support channels

| Channel | Where |
|---------|-------|
| Support email | `support@mountos.io` |
| Web | `https://mountos.io` |
| Downloads / installer | `https://download.mountos.sh` (Cloudflare R2) |

For help, contact `support@mountos.io`. The web presence is on the
`mountos.io` domain; downloads and the install scripts are served from the
`download.mountos.sh` domain (Cloudflare R2). An issue tracker, chat, or community forum URL is not yet
published (see gaps).


<!-- topic: getting-started -->

# Getting Started (Build a System)

This guide takes a deployment from nothing to a mounted volume. You stand up the HUB once, provision an account and a region, bring up the region's services, create a volume, and mount it.

## Before you start

You bring the infrastructure mountOS runs on:

- A Kubernetes cluster (k3s is the reference target) or a single host for a trial.
- A SQL database (MySQL or PostgreSQL wire-compatible): one for the HUB, one per region.
- A vault (HashiCorp, AWS, GCP, or Azure): one for the HUB, one per region.
- At least one S3-compatible or Azure object store reachable from each region (S3, GCS, B2, MinIO, Azure Blob Storage, or on-prem). A region can point at one or many storages, ideally in the same locality, and AWS S3 or Cloudflare R2 work as is.
- DNS for the HUB and for each region.
- An admin Ed25519 key pair. You keep the private seed; the HUB holds the public verification key.

## 1. Stand up the HUB

Deploy `appserv` with its admin database and Hub vault, seeded with the HUB signing key and the admin verification key. The HUB is the single control-plane entry point and the discovery hub. Publish it behind a stable domain; that domain is the discovery URL every client is given, so choose it once and keep it.

## 2. Connect to the Admin API

The HUB is running now, so from here on you administer it through its Admin API. The SDK points at your own `appserv` (the HUB) URL and talks to it; it administers a running HUB, it does not stand one up. Everything below goes through the HUB's Admin API at `/api/v1/*`. Reach it programmatically with the open-source admin SDK (`@mountos-io/admin-sdk` for TypeScript, `github.com/mountos-io/mountos-admin-sdk/go` for Go, `mountos-admin-sdk` for Rust), or run the open-source admin client (a single-page app plus a small backend you run) and drive it from the dashboard.

The Admin API authenticates with an Ed25519 JWT (`sub=mountos:provider`, `aud=mountos/appserv`). The SDK mints the JWT with the admin key and caches it (1 hour TTL, refreshed 5 minutes before expiry), attaching it to every request:

```ts
import { createServerClient } from '@mountos-io/admin-sdk'

const client = createServerClient({
  baseUrl: 'https://hub.example.com',           // the HUB domain
  privateKey: process.env.MOUNTOS_PRIVATE_KEY,  // admin Ed25519 seed (base64)
})
```

The admin client never holds a long-lived service credential. The operator mints a short-lived (60-second) Ed25519 token and hands the operator a login URL (`https://<dashboard>?token=...`); the admin-client backend verifies it, issues a session, and from then on attaches the SDK-minted, cached service JWT as it proxies the Admin API to the HUB.

## 3. Provision an account and a region

An account is the tenant; it owns its regions, users, and storages. Create the account, add a user, then create a region. Creating a region automatically creates its default cluster, named `uno`.

```ts
const { id: accountId } = await client.accounts.create({ name: 'mountOS' })

const { id: userId } = await client.users.add({
  accountId, username: 'jason', email: 'jason@mountos.io', name: 'Jason Ryan',
})

// creating the region auto-creates its default cluster, "uno"
const { id: regionId } = await client.regions.create({
  accountId, name: 'us-east-1', dns: 'us-east-1.example.com',
})
```

These calls write structure only: control-plane records in the admin database. They spin up no servers and create no instances. mountOS is a software suite you run on your own infrastructure, so you provision the region's real servers, database, vault, and object storage yourself, out of band, in the next step. The `uno` cluster exists now but is not ready: it accepts no volumes until a cluster-scoped service registers into it.

## 4. Bring up the region's services

Deploy the in-region services (dataserv, gcserv, blockserv, and any gateways). Each is configured from its environment: `SERVICE_RPC_ADDR` (the HUB's appserv discovery address, which is how the service finds the HUB), the cluster's `REGION_CLUSTER_ID` (the `uno` UUID, resolved against the HUB at startup to numeric cluster and region shard ids; the cluster carries its owning region, so no separate `REGION_ID` is needed), `ADVERTISE_ADDR` (the IP the service advertises so others and clients can reach it), the region's `DB_URL`, and `VAULT_PROVIDER` plus credentials (each service reads its keys from the vault path `mountos/<service>`). On boot, each service connects to the HUB at `SERVICE_RPC_ADDR`, resolves its cluster (and the region it belongs to), then registers itself, advertising its own IP and ports, and heartbeats every 30 seconds. The HUB records every registered node and serves that topology as discovery to all clients. The first registration into `uno` flips it to ready, after which volumes can be assigned.

Each service binary documents its own configuration: run `<service> env` (for example `dataserv env`) to print the exact environment template for that build, with inline docs and defaults, or `<service> env -w .env` to write it to a file. Missing required values fail fast at boot, so use that template rather than copying older samples.

Registration is strict: a service whose `REGION_CLUSTER_ID` is missing, unknown, or references a deactivated cluster refuses to register rather than landing on the wrong cluster. It raises a topology alert and retries every minute until the topology is fixed.

## 5. Create a storage, a volume, and an access key

Point a storage at your object bucket, create a volume on it, and generate an S3-style access-key / secret-key pair scoped to the volume.

```ts
const { id: storageId } = await client.storages.create({
  accountId, regionId, name: 'prod-s3', storageType: 'object',
  providerType: 's3', endpoint: 'https://s3.us-east-1.amazonaws.com',
  bucket: 'mountos-data', region: 'us-east-1',
})

const { id: volumeId } = await client.volumes.create({
  accountId, storageId, name: 'workspace', volumeType: 'general',
})
```

Then generate the volume's access key: `POST /api/v1/volumes/:volumeId/api-keys/generate` with `{ userId }` returns `{ apiKey, apiSecret }`. The access-key id encodes the volume's routing identity, which is how the HUB resolves a client to its owning region and cluster at discovery time. A volume lands on `uno` by default, or on a named cluster passed as `regionClusterId` / `regionClusterUuid`.

## 6. Mount it from a client

Install the `mountos` client and mount the volume, giving it the HUB domain and the volume's access-key / secret-key pair. The client resolves the volume's region and cluster at the HUB (an unauthenticated discovery call, `GET /api/v1/discover/meta?accessKeyId=...`), then connects to that cluster for the session.

```sh
curl -fsSL https://download.mountos.sh | bash   # installs the mountos client
```

The same data is reachable over S3 and WebHDFS with no mount, using the same key. Run `mountos gateway --gateway s3,hdfs` on the host that needs it, and point S3 tooling or the `hadoop-mountos` Java SDK (the `mountos://` URI scheme) at the local endpoint it publishes.

## Operating: live dashboards

Every service and the client ship a built-in dashboard. `<service> dashboard` (for example `dataserv dashboard`) opens a live terminal UI bound to the running process over its control socket; pass `--gui` for the same view as a local browser GUI instead. The client exposes it per mount: `mountos dashboard <mount-path>` (TUI by default, `--gui` for the browser). Use them to watch a node's health, throughput, Raft state, cache occupancy, and live sessions without standing up external monitoring.

## Scaling from here

You now have a HUB, an account, a region with a ready cluster, a volume, and a client mounting it. To grow it:

- Add clusters to a region (`POST /api/v1/regions/:regionId/clusters/create`) to spread that region's volume load across more service pools.
- Add regions for new localities; each is self-contained with its own database and vault, plus the storages its volumes live on (records pointing at S3-compatible or Azure stores in that locality).
- Hand the HUB domain to more clients.

None of this reconfigures existing clients. They only ever know the HUB domain and resolve everything else from it, so adding clusters, adding regions, or moving a volume between clusters is transparent to them.


<!-- topic: admin-sdk -->

# Admin SDK and API

The Admin SDK is the open, public connector to a running deployment's HUB
(https://github.com/mountos-io/mountos-admin-sdk). It administers a HUB that is already up; it
does not stand one up. Everything the dashboard shows, and everything an operator automates,
goes through the same Admin API at `<HUB>/api/v1/*`.

## Authentication

Two key pairs exist, with different jobs:

- The **admin pair** authorizes Admin API calls. A trusted backend holds the private Ed25519
  seed (the admin-client names it `MOUNTOS_PRIVATE_KEY`); `appserv` holds the public half
  as `PROVIDER_VERIFICATION_KEY` in the Hub vault and verifies every request signature
  against it.
- The **dashboard session pair** (`DASHBOARD_SIGNING_KEY` / `DASHBOARD_VERIFICATION_KEY`)
  backs the opaque session token a dashboard browser holds. It never reaches the Admin API.

Each request carries `Authorization: Bearer <JWT>`, an EdDSA (Ed25519) JWT with
`sub=mountos:provider`, `aud=mountos/appserv`, and a `kfp` key-fingerprint claim, the spec
form is `hex(sha256(ed25519_pubkey)[:16])` (the first 16 bytes of the digest, hex-encoded to
32 characters). The SDKs mint and cache the token automatically (1 hour TTL with a 5 minute
refresh margin); hand-sign only if you are building a new language binding. Never put the
admin seed in a browser.

All three client configs also take an optional `dashboardUser`, HMAC-signed into the
`X-MountOS-Dashboard-User` header. Signing requires a dedicated `dashboardHMACKey` (the same
secret appserv holds as `DASHBOARD_USER_HMAC_KEY`); it is a separate secret from the admin
signing key, never the public verification key. The header both attributes calls to the acting
dashboard user and opts into user-level authorization: `role: "user"` restricts the call to
that operator's account scope, while omitting `dashboardUser` runs with full admin scope.

## Packages

TypeScript, Go, and Rust packages are generated from one API spec and expose the same
surface.

TypeScript (`@mountos-io/admin-sdk`):

```ts
import { createServerClient } from '@mountos-io/admin-sdk'

const client = createServerClient({
  baseUrl: 'https://hub.example.com',   // your running HUB (appserv) URL
  privateKey: process.env.MOUNTOS_PRIVATE_KEY!,
})
const { id } = await client.accounts.create({ name: 'mountOS' })
```

Go (`github.com/mountos-io/mountos-admin-sdk/go`):

```go
import sdk "github.com/mountos-io/mountos-admin-sdk/go"

client, err := sdk.NewClient(sdk.Config{
  BaseURL:    "https://hub.example.com",
  PrivateKey: os.Getenv("MOUNTOS_PRIVATE_KEY"),
})
acct, err := client.Accounts.Create(ctx, &sdk.CreateAccountRequest{Name: "mountOS"})
```

Rust (crate `mountos-admin-sdk`):

```rust
use mountos_admin_sdk::{Client, Config, CreateAccountRequest};

let client = Client::new(Config {
    base_url: "https://hub.example.com".into(),
    private_key: std::env::var("MOUNTOS_PRIVATE_KEY").unwrap(),
    ..Default::default()
})?;
let created = client.accounts.create(&CreateAccountRequest {
    name: "mountOS".into(),
    description: None,
    icon_url: None,
    provider_info: None,
}).await?;
```

The packages are versioned together with the API. The npm package ships `SKILL.md`, a
version-matched drop-in agent skill; read it before writing SDK code so method names match
the installed version exactly.

## Resource groups

Every package exposes the same client namespaces (TypeScript casing shown; Go and Rust use
their language's casing). The method lists are highlights, not exhaustive; the full list per
namespace is in `api.md`:

| Namespace | Covers |
| --- | --- |
| `accounts` | create, list, get, edit, lock/unlock, deactivate, updateQuota |
| `users` | add, list, get, bulk (QUERY, id-array resolve), edit, deactivate |
| `regions` | create, list, get, edit, deactivate |
| `regionClusters` | create, list, get, edit, setDefault, setReady, deactivate |
| `clusters` | cross-region listing of all region clusters |
| `storages` | create, list, get, edit, deactivate, testNewBucket, testStorageBucket |
| `volumes` | create, list, get, edit, lock/unlock, activate/deactivate, moveCluster, updateQuota, stats, sizeHistory, generateAPIKeys, revokeAPIKey, revokeAPIKeysByUser, per-volume fork methods |
| `volumeForkTrees`, `volumeForkEntries`, `volumeForkSearches` | fork trees, entries, and searches |
| `auditLogs`, `regionAuditLogs` | account and region audit feeds (cursor pagination) |
| `serviceNodes` | per-region node listing and per-node stats |
| `nodes` | cross-region listing of all registered service nodes |
| `clientSessions` | live and recent client mounts |
| `alerts`, `regionAlerts` | alert feeds |
| `gcWorkerEvents` | per-region GC worker event feed, histogram, and goal list |
| `metrics` | scrape-token minting for Prometheus/VictoriaMetrics |
| `discover`, `dashboard`, `license`, `vault` | discovery, dashboard support, license info plus load/upload and list, vault resync |

Several GET list/read endpoints also accept the HTTP `QUERY` method (RFC 10008) as a
same-handler alias -- appserv's own transport choice for callers with parameter sets too
large or complex for a URL query string. `users.bulk` is QUERY-only, since resolving a
page's worth of ids never fit a query string in the first place. See
`docs/design/query-verb.md` in the SDK repo for when to reach for it.

Do not invent method names. When unsure, list the namespace's methods in the installed
package or check `api.md`.

## REST

The API is served under `/api/v1` on the HUB domain. Routes are action-suffixed
(`/accounts/create`, `/accounts/list`, `/accounts/:accountId`); the full route list and
token claims are in `api.md` in the SDK repo.

```sh
curl https://hub.example.com/api/v1/accounts/list \
  -H "Authorization: Bearer $JWT"
```

Every response uses one envelope (`StandardResponse`); the SDKs unwrap it, raw REST callers
see it:

```
{ "status": "success"|"failure", "message": string, "data"?: object, "errorCode"?: int }
```

Pagination comes in two shapes, both nested in `data`:

- Page-based (accounts, users, storages, volumes): `{ items, pagination: { page, limit, total, totalPages } }`.
- Cursor-based (audit logs, fork trees, searches): `{ items, nextCursor: int64|null }`. Loop until `nextCursor` is null.

Errors carry a numeric `errorCode` (AppServ codes are `1XXXX`, for example `10201
VALIDATION_FAILED`, `10901 SERVICE_UNAVAILABLE`). The TypeScript package surfaces them as
`MountOSError` with `.status` and `.errorCode`.

## Generating a client for another language

`api.yaml` in the repo root is the API spec the three packages and the docs are generated
from (`make gen` regenerates everything; it is mountOS's own spec format with `jwt`,
`errorCodes`, and `enums` blocks, not OpenAPI). For a language with no official package,
either call the REST API directly (see below), or write a new binding against `api.yaml`
using the Go, Rust, and TypeScript generated clients under `go/`, `rust/src/`, and `ts/src/`
as reference implementations, or request one at support@mountos.io.

### Looking up the exact contract without the repo

You don't need repo access to check current method names, request/response shapes, or route
paths -- each installed package carries its own source of truth:

- **npm (`@mountos-io/admin-sdk`)** ships `api.yaml` and `api.md` alongside `SKILL.md` (see
  `package.json` `files`), plus the compiled `dist/*.d.ts` for the exact generated method
  signatures.
- **Go (`github.com/mountos-io/mountos-admin-sdk/go`)** does not carry `api.yaml`/`api.md`
  (the Go module root is `go/`, so repo-root files fall outside it) -- read the generated
  source directly: `client_gen.go`, `resources_gen.go`, `types_gen.go` in the installed
  module cache.
- **Rust (`mountos-admin-sdk` crate)** ships `api.md` (copied in at publish time) plus its own
  generated `src/client_gen.rs` and `src/types_gen.rs`.

All three `*_gen.*` files are generated and carry a "DO NOT EDIT" header; treat them as
read-only reference, never a copy-paste source to hand-edit into a new binding.

## The dashboard relationship

The admin dashboard (https://github.com/mountos-io/mountos-admin-client) is a browser
frontend plus a small backend gateway. The gateway holds the admin seed and signs Admin API
requests through this SDK; the browser holds only a short-lived opaque session token. Anything
the dashboard does is an Admin API call you can reproduce with the SDK or REST.

### Minting a dashboard sign-in token

Sign-in is a short-lived login URL your backend mints, not something the SDK or appserv
returns. Your system generates its own Ed25519 key pair, keeps the private seed, and hands the
dashboard the public half as `PROVIDER2DASHBOARD_VERIFICATION_KEY`. To sign a user in, mint an
EdDSA (Ed25519) JWT carrying the claims below, then redirect the browser to
`https://<dashboard>/?token=<jwt>`.

| Claim | Required | Notes |
| --- | --- | --- |
| `aud` | yes | Must be `mountos/dashboard`. |
| `sub` | yes | Stable user identifier; your external user id is fine. |
| `iat` | yes | Issued-at, used for the freshness check. |
| `role` | yes | Operator role the dashboard maps to capabilities. The built-in dashboard recognizes `superadmin`, `l1admin`, `l2admin`, and `user`. |
| `exp` | recommended | Mint roughly 60 seconds out. The dashboard rejects any token whose `iat` is more than 120 seconds old, allowing 60 seconds of clock leeway. |
| `name` | recommended | Operator display name. |
| `email` | optional | Operator email. |
| `username` | conditional | Required when `role` is `user`. Optional for admin roles, where it serves only for attribution. |
| `account_id` | conditional | Required when `role` is `user`, and pins the operator to that account's scope. |

Admin-level roles are provider-defined and need no matching mountOS user; appserv treats any
role other than `user` as system-admin scope. Only `role=user` must resolve to an existing
active user in the named account (by `username` and `account_id`), and the exchange fails when
no such user exists. A username supplied with an admin role is only found or created under the
system account, never required to pre-exist.

The dashboard verifies the token at `POST /api/auth/exchange` against
`PROVIDER2DASHBOARD_VERIFICATION_KEY`, resolves the user through this SDK, then issues its own
httpOnly session and refresh cookies. The sign-in token authenticates only that one exchange
and never reaches appserv. Keep the minting seed on your backend, never in a browser.


<!-- topic: change-events -->

# Change Events

Every general volume can keep an ordered feed of its changes: `create`,
`delete`, `modify`, and `rename`. The feed is per fork (fork 0 is the main line),
ordered by a per-fork monotonic sequence number, and read with the `mountos event`
command, either as a stream on stdout or served as a local REST API. It exists so
indexers, sync tools, and AI agents can follow a volume without walking the tree.

General volumes only. Iceberg/lake volumes write through the segment plane and bypass
the metadata arena the feed is captured from, so both the client and the server reject
the request explicitly (no silently empty feed).

## Enabling: one volume knob

The feed is off by default. `eventLogRetentionPeriod` (days, 0-30, default 0) on the
volume gates capture AND sets how long events stay readable. At 0 nothing is captured
and no rows accumulate. Set it at `volumes.create` or later via
`PUT /api/v1/volumes/:volumeId/edit` (SDK `volumes.edit`); in Go and Rust the edit
field is a pointer/`Option` so omitting it preserves the persisted value and an
explicit 0 disables the feed.

`gcserv` enforces retention per volume: a cluster-scoped goal deletes `event_log` rows
older than the volume's window, on a roughly 6-hour cadence, in bounded 1000-row
batches.

## Capture semantics

Capture runs on the volume's owner dataserv node, on the hot mutation path, after the
change lands in the in-memory arena:

- Gates on per-volume retention first (0 = no work at all).
- Resolves the absolute path from the arena without allocating.
- Coalesces repeated `modify` events to the same file within a flush window.
- Appends to a sharded, lock-striped buffer; a single background flusher assigns the
  per-fork `seq`, stamps the owner `incarnation_id`, and batch-inserts into the region
  database's `event_log` table about every 3 seconds (default flush interval).

Consequences for consumers: the feed is near real time (seconds, not milliseconds),
a burst of writes to one file surfaces as fewer `modify` events than writes, and an
`incarnation_id` change between consecutive events marks a possible gap (owner restart
or failover), a signal to reconcile the affected subtree if exactness matters.

## Event shape

| Field | Meaning |
| --- | --- |
| `seq` | Per-(volume, fork) monotonic sequence number, the resume cursor |
| `fork_id` | Fork the event belongs to (0 = main) |
| `incarnation_id` | Owner incarnation; a change between consecutive rows marks a possible gap |
| `event_ts` | Capture wall clock in microseconds, the canonical feed time |
| `born_at` | Source attrs `born_at`, correlates with version history |
| `inode` | Inode the event applies to |
| `event_type` | `create`, `delete`, `modify`, `rename` |
| `type` | File type: `file`, `dir`, `symlink`, `blkdev`, `chrdev`, `fifo`, `socket` |
| `size` | Size at capture (0 renders as `-` in table output) |
| `path` | Full path; for `rename` the destination |
| `from_path` | Rename source path, present only on renames |

## Stream mode (stdout)

`mountos event` authenticates with the volume's access-key pair and resolves the owning
cluster at the HUB, exactly like a mount. Credentials: `-a <access-key-id>` plus `-s`
(prompts for the secret), or `MOUNTOS_ACCESS_KEY_ID` / `MOUNTOS_SECRET_ACCESS_KEY`; the HUB
domain comes from `--discovery-url` or `MOUNTOS_DISCOVERY_URL`.

```sh
mountos event -a AKIA... -s                            # everything retained, oldest first
mountos event -a AKIA... -s --since 7d --follow        # last 7 days, then tail
mountos event -a AKIA... -s --path-prefix /docs --json # one subtree, one JSON object per line
mountos event -a AKIA... -s --start-seq 100            # sequence floor instead of time
```

- `--since`: time floor; relative (`7d`, `2h`) or timestamp (`"2025-01-01 14:30"`).
- `--start-seq`: sequence floor (events with `seq` greater than the value).
- `--path-prefix`: server-side subtree filter on a path-component boundary (`/docs`
  matches `/docs/x`, not `/docs2`; filtering happens in SQL so non-matching rows are
  never transferred).
- `--follow`: tail, polling every 5 s. Without `--since`/`--start-seq` it seeks to the
  live end first instead of replaying retained history.
- `--json`: one JSON object per event (same shape as the REST API); `--no-header`
  suppresses the table header.
- Fork selection: the global `--fork-name` flag (empty = main).

Pages are fetched 1000 at a time; the server clamps any requested page size to
[100, 1000].

## HTTP API mode (--http)

`mountos event ... --http` starts a local REST server instead of streaming; stream
flags are ignored and callers control pagination. The process daemonizes by default
(`--foreground` keeps it attached), binds `127.0.0.1`, and uses an OS-assigned port
unless `--http-port` is set. Exposure beyond loopback requires TLS:
`--http-no-loopback` is rejected without `--http-cert` + `--http-key`.

Auth is a signed challenge; the secret never crosses the wire:

```http
POST /v1/auth
{
  "access_key_id": "<key>",
  "timestamp": <unix-seconds, within 5 minutes>,
  "nonce": "<random hex, single use>",
  "signature": "<hex(HMAC-SHA256(secret, key:timestamp:nonce))>"
}
-> { "token": "<bearer>", "expires_in": 3600 }
```

```http
GET /v1/events
Authorization: Bearer <token>
Initial query params: since, start_seq, path_prefix, limit (1-1000, default 1000; values under 100 are served as 100), seek_to_end=true
Continuation: continuation_token=<opaque> (encodes cursor + filters; limit may override per page)
-> { "events": [...], "continuation_token": "<opaque>", "has_more": <bool> }
```

The server serves the fork it was started against (the global `--fork-name` flag at
launch). A `fork_id` query param is accepted and echoed through the continuation
token but does not change which fork is served; do not rely on it.

`seek_to_end=true` returns a continuation token at the live end with no events (the
API form of tail-from-now). Bearer tokens expire after 1 hour; re-auth with a fresh
nonce. Timestamps outside the 5-minute drift window and reused nonces are rejected.

## Consumer checklist

- Resume by storing the last processed `seq` (or the opaque continuation token).
- Poll at a cadence well inside `eventLogRetentionPeriod` so the prune floor never
  passes the cursor.
- Treat an `incarnation_id` change as a possible gap; reconcile if exactness matters.
- Expect coalescing: the feed reports that a file changed, not every write.
- Storage note: the feed lives in the region database (`event_log`, partitioned,
  indexed for time reads and retention deletes). High-churn volumes with long windows
  grow the table accordingly; size retention to what consumers actually replay.


<!-- topic: troubleshooting -->

# Troubleshooting

mountOS diagnoses itself. Reach for the built-in tooling first, then the logs. The
log reference below maps verbatim error strings to fixes for the cases an agent greps.

## Start with mountos check

`mountos check` verifies the mount backends on the machine it runs on and prints the
fix for every failing check, including the exact commands (extension re-registration,
system log queries) and log locations.

- macOS: FSKit (OS >= 26, framework, mountOSFSKit.app, extension registered + enabled),
  macFUSE (installed, version), NFS (mount_nfs, NFSv4.1), SMB (mount_smbfs).
- Linux: io_uring readiness for FUSE (kernel >= 6.14, CONFIG_FUSE_IO_URING,
  fuse.enable_uring, syscall availability).
- Windows: mountosio.sys kernel driver, log location
  `%LOCALAPPDATA%\mountOS\mountos.log`.

Flags: one per backend (`--fskit`, `--macfuse`, `--nfs`, `--smb`, `--io-uring`)
and `--json` for machine-readable output. Client logs
live under `~/.mountOS/logs/` (macOS/Linux). The client's full reference is built in:
`mountos man` prints the manual page (nroff-formatted when available), and every
command documents itself with `--help`.

Live state is always inspectable: `<service> dashboard` on a node,
`mountos dashboard <mount-path>` on a client (`--gui` for the browser), and the admin
dashboard's alerts and nodes views.

## Let an AI agent troubleshoot

Every mount exposes a local, read-only MCP connector (stdio, JSON-RPC). Register it
once with `mountos mcp install` (Claude Desktop, Claude Code, Codex, Gemini;
`mcp status` / `mcp uninstall` manage it), or configure manually with
`{"mcpServers":{"mountOS":{"command":"mountos","args":["mcp"]}}}`. An attached agent
queries running mounts directly: instances, identity, live stats and throughput,
recent deletions, recovered (lost-and-found) files, configuration, diagnostics, file
version history, inode resolution, backend capability checks, and the CLI commands
that operate a mount. It never reads file contents and never mutates state, safe on
production mounts. Pair it with the published agent skill, skill.md at the site root.

## A service will not register

Symptom: the cluster stays not ready, or a deployed service never appears in the
topology. The service keeps running. It logs
`discovery registration deferred; will retry` with `retry_interval: 1m0s` and retries
every minute, so the service's own log is the place to look (the errors below are
raised client-side at the resolve stage, before a register request is sent). When a
register request reaches the HUB and is rejected there, the HUB additionally raises
a critical topology alert titled `service node rejected: <service>/<node>` in the
dashboard's alerts view.

| Error | Fix |
| --- | --- |
| `REGION_CLUSTER_ID is required for discovery registration` | Set the env var to the cluster's `exportId` UUID (from the region's Clusters list); the region is derived from it |
| `resolve region cluster <uuid>: cluster is deactivated` | Reactivate the cluster (dashboard or Admin API) or point at an active one |

Admin-plane fixes (for example reactivating a cluster) are picked up on the next
retry, no restart. Environment fixes need a service restart.

## A service exits at boot

Boot failures are deliberate and loud.

| Error | Fix |
| --- | --- |
| `missing required environment variables: ...` | Run `<service> env` for that build's annotated template, set the listed vars |
| `license signature verification failed` | The HUB rejected an invalid payload on upload, or a service rejected a corrupted payload on fetch; re-upload a valid signed payload through the admin client or SDK |
| `hashicorp vault auth: ...` | Vault unreachable or AppRole credentials wrong; verify from the service host |
| `ED25519_SIGNING_KEY not found in secrets` | The service's vault entry at `mountos/<service>` (or `mountos-<prefix>/<service>` if `VAULT_RESOURCE_PREFIX` is set) is incomplete; see the secrets topic |
| `VAULT_RESOURCE_PREFIX ... is invalid` | The prefix must be 1-11 lowercase alphanumeric/hyphen characters, not starting or ending with a hyphen; see the secrets topic |

A boot no longer fails on a missing license file. Services boot without one and
learn the license state from the HUB heartbeat. A healthy service logs its
license state sourced from the heartbeat (`[license] Valid until <date> UTC
(licensee: ...)`); the exact wording may differ from the old file-sourced line.

Licenses live in the admin database on the HUB. Each is a signed JWT-like payload,
keyed by the SHA1 of its body, so a duplicate upload is ignored and a copy never
doubles a quota. The HUB verifies each payload on upload and rejects the batch if
any is invalid, computes the active set, and sums its capacity (storage, nodes,
volumes, users, accounts, regions). With no active license the deployment runs on
the Free tier, up to 10 TiB. Add capacity through the admin client or SDK by
uploading a license. Licenses of one identity stack, so an add-on or renewal adds
to the active capacity and an expired license drops out. The change reaches every
service within one heartbeat (about 30 seconds). A payload with a future start date stays dormant until that date,
then activates on its own.

Every service gets its license state from the heartbeat, a small status (free or
paid) plus a change marker. When the marker changes the service fetches the full
signed payloads from the HUB, verifies them locally against its built-in verifier
key, and caches them. Services never read a license file. Offline, if the HUB is
unreachable a service keeps its last known license: a real license rides its own
validity plus grace, and the Free tier is good for up to 30 minutes since the last
HUB contact, after which the service degrades to its existing no-license behavior.
Expiry is evaluated locally against each cached payload, rechecked every 24 hours
when valid and every 10 minutes under grace.

An expired license does not stop a running deployment immediately. The lifecycle
has three stages. During the grace period the service keeps serving normally and
logs `[license] WARNING: License expired, grace period until <date> UTC. Contact:
<contact>`. After grace it serves read-only through the expired-access window,
logged as `In expired access (read-only) mode until <date>`. In read-only mode
every data-facing service (the hub, metadata server, block storage, and the S3
and WebHDFS gateways) rejects writes while still serving reads, and background
garbage collection pauses; the same read-only protection stays in force once the
expired-access window itself ends, and a fresh start in that state is refused.

## db install and migrate

| Error | Fix |
| --- | --- |
| `db: DB_URL not set` | Database settings come from the service's vault entry, not the env; set `DB_URL` and `DB_DIALECT` there |
| `migrate: postgresql/admin already installed; use 'db migrate' to upgrade` | `db install` is for a fresh database; existing ones upgrade with `db migrate` (idempotent) |
| `db: fetch schema manifest for ...` | The schema downloads from download.mountos.sh; allow the endpoint or pre-fetch with `db download` |

`DB_DIALECT` takes `postgresql` or `mysql`. Set it exactly.

## A client cannot mount

| Error | Fix |
| --- | --- |
| `--discovery-url or MOUNTOS_DISCOVERY_URL is required` | Give the client the HUB domain |
| `discovery: appserv failed (...) and no cache available` | HUB unreachable and this client never discovered before; check DNS/TLS toward the HUB. A previously mounted client rides out a HUB outage on cached topology |
| `access-id must be exactly 20 characters, got 19` (the secret must be exactly 40) | Truncated paste or trailing whitespace in the key pair |
| `credentials not provided: use -access-id and -secret-access-key flags or set MOUNTOS_ACCESS_KEY_ID and MOUNTOS_SECRET_ACCESS_KEY environment variables` | Provide the volume's key pair |
| `no endpoints available` | The volume's cluster has no registered service; bring up the region services (see registration above) |
| `authentication timeout` | Printed by `mountos fork` / `mountos event` when the session never authenticates (the mount path prints the connect checklist instead); verify the secret and that nodes show in the dashboard's nodes view |
| `--fork-name "dev" does not exist` | `mountos fork list`, then mount an existing fork name |
| `iceberg volume cannot be mounted writable` | Iceberg volumes mount read-only; writes go through the catalog |

A failed connect prints its own checklist: service running and accessible, access
key/secret correct, network connectivity.

## A gateway rejects requests

| Error | Fix |
| --- | --- |
| `volume type not supported by this gateway` | The S3 and WebHDFS gateway serves general volumes only; Iceberg volumes are reached through the Iceberg catalog, which the client starts automatically for that volume type |
| `--gateway-no-loopback requires --gateway-cert and --gateway-key (no plaintext off-host)` | Off-host binding requires TLS, both files. Same rule as the event server's `--http-no-loopback` |
| `--gateway-key ... has loose permissions 0644; expected 0600 or stricter` | `chmod 600` the key file |

## Change-event errors

The HTTP API's 401 bodies and the command's own startup errors:

| Error | Fix |
| --- | --- |
| `timestamp outside acceptable window` | Auth timestamp must be within 5 minutes of the server clock; sync clocks |
| `nonce already used` | Nonces are single use; generate a fresh one per auth request |
| `invalid signature` | HMAC-SHA256 over `key:timestamp:nonce` in that order, keyed with the secret |
| `invalid or expired bearer token` | Bearers live 1 hour; re-authenticate |
| `event feed is available only for general volumes (this volume is "iceberg")` | Printed when `mountos event` starts; the feed exists for general volumes only |

## Listing a huge directory

A mountOS directory read returns entries together with their attributes, so list
flags that print per entry stream at full speed: `ls -ioFs1 <dir>`. Plain `ls`
buffers the whole directory for column layout before printing anything. The inode
column feeds `resolve_inode` and the version/recovery tooling (see the client's MCP
`get_tips`).

## Where to look next

Full contracts are in the getting-started and change-events topics. Anything the
tooling does not explain goes to support@mountos.io.