# mountOS Skill: Provision and bring up a deployment

> Take a deployment from a fresh HUB to a region cluster that is ready to hold volumes.

This is the first bring-up. You stand up the one HUB, connect to its Admin API, create
the tenant (an account) and its users, create a region, then boot the region's services so
its default cluster flips to ready. Stop before creating volumes and keys. That is the
volumes skill (https://mountos.io/skills/volumes.md). The order below is load-bearing: skip a step
and the next one surfaces it immediately (most fail loudly), and a mis-registered service
raises a critical topology alert and retries every minute until the topology is fixed.

## Mental model (do not violate)

- The **HUB** is `appserv`, one per deployment, multi-tenant. It owns the admin database and
  the Hub vault, serves the Admin API at `<HUB>/api/v1/*`, and answers client discovery.
  Its domain is the discovery URL every client is given. Choose it once and keep it.
- An **account** is the tenant. It owns its regions, users, and storages.
- A **region** belongs to one account and owns exactly one database and one vault; it can
  hold one or many storages, region-scoped records that each point at an S3-compatible or
  Azure store. A volume lives in one region on exactly one storage; its data never crosses a
  region boundary at serving time.
- A **region cluster** partitions volume load inside a region. It shares the region's one
  database and one vault. It is not a tenant boundary and gets no DB or vault of its own.
  `dataserv`, `gcserv`, `blockserv`, and the gateways are cluster-scoped.

## 1. Stand up the HUB

Deploy `appserv` with its admin database and Hub vault. The Hub vault holds appserv's own
secret (DB settings + its Ed25519 keypair), the `service-verifiers` set, and
`PROVIDER_VERIFICATION_KEY` (the admin Ed25519 public key, used to verify Admin SDK
JWTs). appserv runs as the reserved hub region and self-registers into `hub_nodes`,
heartbeating every 30s; unlike the cluster-scoped services it carries no `REGION_CLUSTER_ID`.

Required env (from config validation): `DB_URL` (admin DB), `ED25519_VERIFICATION_KEY`,
`PROVIDER_VERIFICATION_KEY`, plus `HTTPS_PORT`, `TLS_CERT_FILE`, `TLS_KEY_FILE`, and the
`VAULT_PROVIDER` block. Publish appserv behind a stable domain (TLS). That domain is
`<HUB>` everywhere below.

mountOS ships binaries, not images. Pull each, pinning a version:
`curl -fsSL https://download.mountos.sh | bash -s -- --pkg mountos-appserv --version X.Y.Z`
(region services use `mountos-dataserv`, `mountos-gcserv`, and optionally `mountos-blockserv`;
`--list` lists packages). Topology and deploy
samples are in the deployment topic (https://mountos.io/ai/topics/deployment.md); secret layout is in
https://mountos.io/ai/topics/secrets.md.

**Create the schema.** Set `DB_DIALECT` in the vault alongside `DB_URL` (`postgresql` or
`mysql`; use the PostgreSQL dialect unless you already run a MySQL-wire store. PostgreSQL batches more efficiently, and the schema ships for both), then
run `appserv db install` once. It reads `DB_URL` + `DB_DIALECT` from the vault, downloads the
matching-dialect admin schema (cached under `~/.mountOS/downloads`), and applies it. Later
upgrades use `appserv db migrate` (idempotent, applies only the new migrations).

You provide the admin Ed25519 key pair yourself. Keep the 64-byte private seed
(base64, standard encoding, 88 chars); seed only the 32-byte public key into appserv as
`PROVIDER_VERIFICATION_KEY` (44 chars).

## 2. Connect to the Admin API

The HUB is up now, so from here you administer it through its Admin API; the SDK points at
your own `appserv` (the HUB) URL and does not stand a HUB up. Everything below goes through
`<HUB>/api/v1/*`. The API authenticates with an Ed25519 JWT
(`sub=mountos:provider`, `aud=mountos/appserv`). Use the open-source admin SDK
(`@mountos-io/admin-sdk` for TypeScript, `github.com/mountos-io/mountos-admin-sdk/go` for
Go, `mountos-admin-sdk` for Rust); it mints an Ed25519 JWT with the admin key and caches it
(1-hour TTL, refreshed 5 minutes before expiry), sending the cached token on every request.

```ts
import { createServerClient } from '@mountos-io/admin-sdk'

const client = createServerClient({
  baseUrl: 'https://hub.example.com',          // the HUB domain, <HUB>
  privateKey: process.env.MOUNTOS_PRIVATE_KEY, // admin Ed25519 seed (base64, 88 chars)
})
```

The SDK exposes the full resource surface: `accounts`, `users`, `regions`, `regionClusters`,
`storages`, `volumes` (including `generateAPIKeys` / `revokeAPIKey`), fork trees, audit
logs, nodes, client sessions, alerts, license, and vault. The per-namespace method list is
in https://mountos.io/ai/topics/admin-sdk.md. When a method is missing on an older
package, call the REST path directly with the same signed JWT.

Equivalent raw call, if you sign the JWT yourself (payload also needs `iat`, `nbf`,
`exp=iat+3600`, `jti`, `scope: "service"`, and `kfp` = `hex(sha256(pubkey)[:16])`):

```sh
curl -sS https://hub.example.com/api/v1/accounts/list \
  -H "Authorization: Bearer $PROVIDER_JWT"
```

**Never put the admin private key in a browser.** For a human-driven UI, the operator
mints a short-lived (~60s) Ed25519 token and hands the operator a login URL
(`https://<dashboard>?token=...`). The `mountos-admin-client` Bun gateway verifies it,
issues a session cookie, and from then on reuses a cached service JWT for proxied requests
(1-hour TTL, refreshed 5 minutes before expiry). The
private seed stays server-side. See https://mountos.io/ai/topics/components.md for the dashboard
auth chain.

## 3. Create the account and its users

An account is the tenant. Create it, then add users. You need at least one user id later to
generate volume access keys.

```ts
const { id: accountId } = await client.accounts.create({ name: 'mountOS' })

const { id: userId } = await client.users.add({
  accountId,
  username: 'jason',
  email: 'jason@mountos.io',
  name: 'Jason Ryan',
})
```

REST equivalents: `POST /api/v1/accounts/create {name}` and
`POST /api/v1/users/add {accountId, username, email, name}`. Both return `{ id }` (int64).
Ids are numeric; keep `accountId` and `userId` for the steps below.

## 4. Create a region (auto-creates cluster "uno")

A region binds to the account and carries a name plus a DNS entry. Creating it
automatically inserts the region's single default cluster, named `uno`
(`default_cluster=true`, `is_active=true`, `is_ready=false`).

```ts
const { id: regionId } = await client.regions.create({
  accountId,
  name: 'us-east-1',
  dns: 'us-east-1.example.com',
})
```

REST: `POST /api/v1/regions/create {accountId, name, dns}` returns `{ id }`. All three
fields are required.

This writes **structure only**: control-plane records in the admin database. No servers or
instances are created. mountOS is a software suite the operator self-hosts, so you provision
the region's real database, vault, object store, and services yourself, out of band (step 5).

`uno` exists now but is **not ready**: it accepts no volumes until a cluster-scoped service
registers into it (step 5 flips it). Fetch it to get the UUID you pass as
`REGION_CLUSTER_ID`. The list is page-paginated (`limit` default 20), so the response is
`{ items, pagination }`, not a flat array. There is no SDK method for this; call the REST
path:

```sh
curl -sS https://hub.example.com/api/v1/regions/$REGION_NUMERIC_ID/clusters/list \
  -H "Authorization: Bearer $PROVIDER_JWT"
# data: { items: RegionCluster[], pagination: { page, limit, total, totalPages } }
# pick the entry with defaultCluster=true; uno.isReady is false until a service registers
```

REST: `GET /api/v1/regions/:regionId/clusters/list`. Services need only the cluster UUID,
which carries its owning region:

- **`REGION_CLUSTER_ID`** = the **RegionCluster** (`uno`) object's `exportId`.

There is no separate region env var: appserv resolves the cluster's owning region (and
account) at startup. The numeric `id` and the `exportId` UUID are different things: services
take the **UUID** (`REGION_CLUSTER_ID`); the Admin API path takes the numeric `id`.

Adding more clusters later is the day-2 skill (https://mountos.io/skills/operate.md), via
`POST /api/v1/regions/:regionId/clusters/create {name}`.

## 5. Bring up the region's services

Provision the region's own infrastructure first: a regional database and a Region vault.
Object storage needs no provisioning here: a storage
is a record pointing at an existing S3-compatible or Azure store (one or many per region,
ideally in the same locality; AWS S3 and R2 work as is), created in the volumes skill. The Region vault holds each
regional service's own secret, a replicated copy of `service-verifiers`, an `api-master`,
and (written by the services) `s3creds` and `volcreds`. Replicating `service-verifiers`
from the Hub vault to every Region vault is the operator's job (see
https://mountos.io/ai/topics/secrets.md); cross-service JWT verification fails without it.

Now deploy the cluster-scoped services with the cluster UUID (`REGION_CLUSTER_ID`) and the region's
DB and vault. The cluster carries its owning region, so the region shard is derived from it and no separate region UUID is set. Each boots, resolves the cluster UUID to numeric cluster and region shard ids against the HUB,
registers with its advertise/RPC (and for `dataserv`, raft) addresses, and heartbeats every
30s.

Size `dataserv` first: 3 or 5 instances per cluster (odd for quorum; more only adds quorum
overhead), on machines with large RAM and high network bandwidth. `METAENGINE_ARENA_SIZE`
dedicates the metadata arena upfront (min 128MB; a slot is 192 bytes plus a name allowance,
so plan for about five million files per GiB, and names over the allowance push memory
up a bit); size it to the metadata working set. Networking: every service
advertises a public IPv4 (`ADVERTISE_ADDR`, an IP, not a hostname). No region service needs
DNS or TLS; `dataserv`, `gcserv`, and `blockserv` take neither (`blockserv` is
reached by direct IP, with a transport revamp pending). Discovery hands clients the
addresses, and `gcserv` stays hidden inside the region with health read through the HUB's
nodes view.

**Create the regional schema** once per region, before the data services serve traffic:
`dataserv db install` (reads `DB_URL` + `DB_DIALECT` from the Region vault, downloads the
matching-dialect data schema, applies it). `gcserv` shares the same regional database, so one
`db install` per region covers both. Later changes apply with `db migrate`.

| Service | Role | Required env (beyond `SERVICE_RPC_ADDR`, vault block; the license arrives from the HUB heartbeat, no per-service file) |
| --- | --- | --- |
| `dataserv` | Meta backbone (Raft owner per `(volume, fork)`) | `REGION_CLUSTER_ID`, `DB_URL` (regional), `ED25519_SIGNING_KEY`, `ED25519_VERIFICATION_KEY` |
| `gcserv` | GC/cleanup. Runs as its own pool per cluster | `REGION_CLUSTER_ID`, `DB_URL` (regional), `ED25519_VERIFICATION_KEY` |
| `blockserv` (optional) | Block-storage byte plane (block protocol, not S3), proxy or actual storage on your own block devices. One block volume of a storage; HA storages run 2 to 3 across distinct clusters | `REGION_CLUSTER_ID`, `BLOCK_VOLUME_ID`, `MOUNTOS_DISCOVERY_URL` (dataserv `host:port`), `STORAGE_MOUNT_ROOT_PATH` |

S3 and WebHDFS need nothing here. They are started per volume from the client
(`mountos gateway --gateway s3,hdfs`) wherever the protocol surface is wanted.

`SERVICE_RPC_ADDR` is the **appserv internal RPC** address used for registration, heartbeat, and
discovery (format `host:port`). It is a separate internal RPC port from appserv's HTTPS discovery
URL; do not assume it equals `HTTPS_PORT`. `MOUNTOS_DISCOVERY_URL` differs by consumer: for
`blockserv` it is a raw `host:port` for the internal protocol to `dataserv` (default
`6464`, no scheme); for client discovery it is the appserv HTTPS URL. Do not cross them.
Each node resolves its advertise IPv4 from `ADVERTISE_ADDR`, else `CLOUD_PROVIDER` IMDS,
else auto-probe; boot fails if none resolves. Full env surface:
https://mountos.io/ai/topics/configuration.md. Each binary also prints its own current template: run
`<service> env` (stdout) or `<service> env -w .env` (file). Prefer that over copying samples;
it matches the running build exactly.

```sh
# blockserv unit env (excerpt)
REGION_CLUSTER_ID=2a91...                      # uno (RegionCluster) exportId (UUID); region derived from it
SERVICE_RPC_ADDR=hub.example.com:7443          # appserv internal RPC host:port (example port; not HTTPS_PORT)
MOUNTOS_DISCOVERY_URL=dataserv.us-east-1.internal:6464  # dataserv host:port, no scheme
STORAGE_MOUNT_ROOT_PATH=/var/lib/mountos/blocks  # block-device cache mount (XFS, mounted rw,async,noatime,nodev,noexec,nosuid)
VAULT_PROVIDER=hashicorp
VAULT_HASHICORP_ADDRESS=https://vault.us-east-1.internal:8200
VAULT_HASHICORP_ROLE_ID=...
VAULT_HASHICORP_SECRET_ID=...
```

**Block device.** The cache at `STORAGE_MOUNT_ROOT_PATH` holds many small files, all opaque
blob/object parts. Format it with XFS and mount it `rw,async,noatime,nodev,noexec,nosuid` in
`/etc/fstab`. `rw` and `async` are the defaults: it must be writable, and buffered I/O is
right for a cache whose durable floor is object storage (`sync` would cripple it). `noatime`
keeps reads from turning into writes (write amplification on the SSD) and covers directories
too. `nodev`, `noexec`, and `nosuid` harden a data-only mount: no device files, nothing
executed, setuid bits ignored. XFS allocates inodes across the whole device by default
(`inode64`).

**Registration is strict.** A service whose `REGION_CLUSTER_ID` is empty, unknown, or
references a deactivated cluster starts but refuses to register rather than landing on the
wrong cluster; it raises a critical topology alert and retries every minute until the
topology is fixed. This is a guard, not an error to work around. Fix the UUID.

**The first successful registration of a cluster-scoped service flips `uno` to
`is_ready=true`.** After that, volumes can be assigned to it. This first-registration
mechanic is detailed here; the day-2 skill cross-links back rather than repeating it.

## 6. Verify the cluster is ready

```sh
curl -sS https://hub.example.com/api/v1/regions/$REGION_NUMERIC_ID/clusters/list \
  -H "Authorization: Bearer $PROVIDER_JWT"
# data.items: the uno entry should show isReady=true, isActive=true

curl -sS "https://hub.example.com/api/v1/regions/$REGION_NUMERIC_ID/nodes" \
  -H "Authorization: Bearer $PROVIDER_JWT"   # registered dataserv/gcserv/blockserv nodes
```

The list response is `{ items: RegionCluster[], pagination }`. Find the entry with
`defaultCluster=true` and confirm `isReady=true` and `isActive=true`.

If `isReady` stays `false`: no cluster-scoped service has registered. Check service logs for
a registration error (UUID mismatch, empty `REGION_CLUSTER_ID`, vault unreachable, or no
advertisable IPv4). `GET /api/v1/regions/:regionId/nodes` lists what registered. As a last
resort the Admin API exposes
`POST /api/v1/regions/:regionId/clusters/:clusterId/set-ready {ready: true}`, but prefer
fixing the service so it registers. `set-ready` does not stand up a data plane.

## Where you are now

You have a HUB, an account with a user, a region whose default cluster `uno` is ready, and
the in-region services running. Next, create a storage, a volume, and an access key, then
mount it.

## Read next

- Index: https://mountos.io/skill.md
- Create storages, volumes, and access keys: https://mountos.io/skills/volumes.md
- Mount, S3, WebHDFS, CSI: https://mountos.io/skills/integrate.md
- Day-2: add clusters, add regions, move volumes: https://mountos.io/skills/operate.md
- Full topics: https://mountos.io/ai/topics/getting-started.md, https://mountos.io/ai/topics/deployment.md, https://mountos.io/ai/topics/secrets.md, https://mountos.io/ai/topics/configuration.md
- Installer and downloads https://download.mountos.sh, support support@mountos.io