# mountOS Skill: Operate a running deployment (day 2)

> Scale clusters and service pools, move volumes, rotate keys, watch health, and run the volume lifecycle on a live mountOS deployment.

This skill is day 2. The HUB is up, an account and region exist, the region's
default cluster (`uno`) is ready, and volumes are serving. You drive everything
through the HUB Admin API at `<HUB>/api/v1/*`, signed with the admin Ed25519
key (`sub=mountos:provider`, `aud=mountos/appserv`). For initial provisioning and
region-service bring-up see https://mountos.io/skills/provision.md; for creating volumes
and access keys see https://mountos.io/skills/volumes.md. Connect once:

```ts
import { createServerClient } from '@mountos-io/admin-sdk'
const client = createServerClient({ baseUrl: '<HUB>', privateKey: process.env.MOUNTOS_PRIVATE_KEY })
```

The SDK wraps the common control-plane calls (`accounts`, `users`, `regions`,
`storages`, `volumes`, plus `serviceNodes.list` and `vault.resync`). Region-cluster
and volume-move operations are REST-only. When a method is not exposed on the SDK,
call the REST path directly with the same signed JWT.

Ground rules that keep you correct: a region owns one database and one vault, and
holds one or many storages, each a record pointing at an S3-compatible or Azure store. A
region cluster is a volume-load partition inside that region. It
shares the region's one DB and one vault and is not a tenant boundary. A volume
lives in exactly one region and is served by exactly one cluster. None of the
operations below touch clients. Clients only ever know the HUB domain and
re-resolve from it.

## 1. Scale a region by adding clusters

A cluster spreads one region's volume load across more cluster-scoped service
pools (`dataserv`, `gcserv`, `blockserv`). Add one, deploy its services,
then start placing volumes on it. Use the SDK's `regionClusters` namespace
(create, list, get, edit, setDefault, setReady, deactivate) or the REST paths:

```
POST /api/v1/regions/:regionId/clusters/create   { "name": "dos" }  -> { "id" }
```

A new cluster is `isReady=false` and accepts no volumes until its first
cluster-scoped service registers into it. Bring up `dataserv` (+ `gcserv`,
`blockserv`) with `REGION_CLUSTER_ID` = the new cluster's UUID (the
region is derived from it). The registration model (strict
guard, first registration flips `isReady`) is covered in depth in
https://mountos.io/skills/provision.md; `uno`, the auto-created default, follows the same
rule.

### Cluster lifecycle

| Operation | REST | Effect |
| --- | --- | --- |
| Create | `POST /regions/:regionId/clusters/create` `{ "name" }` | New partition, `isReady=false`, not default. Returns `{ id }` |
| List | `GET /regions/:regionId/clusters/list` | `RegionCluster[]` with `defaultCluster`, `isReady`, `isActive` |
| Get | `GET /regions/:regionId/clusters/:clusterId` | One `RegionCluster` |
| Set default | `POST /regions/:regionId/clusters/:clusterId/set-default` | New volumes with no explicit cluster land here. Exactly one default per region |
| Set ready | `POST /regions/:regionId/clusters/:clusterId/set-ready` `{ "ready": bool }` -> `{ id, ready }` | Force-toggle volume acceptance |
| Edit | `PUT  /regions/:regionId/clusters/:clusterId/edit` `{ "name" }` | Rename (unique within region) |
| Deactivate | `POST /regions/:regionId/clusters/:clusterId/deactivate` | Stops accepting volumes; services on a deactivated cluster refuse to register |

Move volumes off a cluster before deactivating it (see section 2). A cluster that
is `isReady=false` or `isActive=false` blocks new volume assignment. A deactivated
cluster also rejects service registration: the service stays up, raises a critical
topology alert, and retries every minute rather than landing a node in the wrong
place.

## 2. Move a volume between clusters

Rebalance load, or drain a cluster you intend to deactivate, by moving volumes.
The target must be a ready, active cluster in the same region as the volume. A
cluster in a different region is rejected. `move-cluster` is REST-only; there is
no SDK wrapper. Call the endpoint with the signed JWT:

```
POST /api/v1/volumes/:volumeId/move-cluster   { "targetClusterId"?: int64, "targetClusterUuid"?: string }
  -> { "id", "sourceClusterId", "targetClusterId", "handoverUntil": int64 }
```

Omit the target to move the volume to the region's current default cluster.

A move is not an instant cutover. `handoverUntil` is a unix-seconds deadline:

1. The source cluster keeps serving the volume and stamps a cluster hint on its
   responses until `handoverUntil`.
2. Clients carry a volume-to-owner topology cache; the hint primes them to
   re-resolve to the destination.
3. After the deadline the destination cluster takes ownership. A background
   handover sweeper clears the transient state. Clients re-resolve at the HUB
   (`GET /api/v1/discover/meta?accessKeyId=...`) and connect to the new cluster.

Do not deactivate the source cluster until every moved volume's window has
expired. Verify placement with `GET /api/v1/volumes/list?regionClusterId=<id>`
before tearing anything down.

## 3. Scale the service pools

Each pool scales independently inside its cluster. Region-wide singleton work
stays unpartitioned; only cluster-scoped services shard by cluster.

| Pool | Scope | How to scale | Notes |
| --- | --- | --- | --- |
| `dataserv` | cluster | Run a Raft cluster of 3 or 5 nodes per cluster | Ownership is per `(volume, fork)`; one owner serves a fork. Keep `RAFT_PORT` fixed across restarts. Odd count for quorum; more only adds quorum overhead |
| `blockserv` | cluster | Run 1 (Single) or 2 to 3 (HA) active-active block volumes per storage, each in a distinct cluster; each advertises its own public IP and self-registers (direct IP, no DNS) | Stateful byte plane (block protocol, not S3) with a local disk cache that syncs to the backing store; drain a block volume before disposing it. Add more block storages for more capacity |
| `gcserv` | cluster | Its own pool per cluster | Size the pool to the background workload (compaction, key alignment); it never sits on the request hot path |

S3 and WebHDFS are not a pool. They run inside the `mountos` client
(`mountos gateway --gateway s3,hdfs`), one process per volume, on whichever host
needs the protocol surface. Scale them by running more client processes, not by
adding server nodes.

New pool nodes self-register with the HUB and heartbeat every 30s. Registration
is strict: an empty or unknown `REGION_CLUSTER_ID`, or a deactivated cluster,
fails registration with a clear error (see provision.md for the full
guard). Confirm a node joined with the nodes list (section 5). The `dataserv`
arena is `METAENGINE_ARENA_SIZE` (min 128MB, allocated upfront at start;
a slot is 192 bytes plus a name allowance, so plan about five million hot metadata records
per GiB; names over the allowance push memory up a bit).

## 4. Vault and key rotation

The deployment has a Hub vault (appserv's secrets + the source-of-truth
`service-verifiers`) and one Region vault per region (regional services' secrets,
a replicated `service-verifiers`, `api-master`, `s3creds`, `volcreds`). All
clusters in a region share that region's one vault. mountOS does not sync vaults;
**replicating `service-verifiers` from the Hub vault to every Region vault is your
responsibility** so cross-service JWT verification works.

| Secret | Rotate | Owner | Downtime |
| --- | --- | --- | --- |
| Service Ed25519 keys (per service) | Weekly/monthly, policy-driven | operator | None: last two keys cached |
| `api-master` (per region) | On demand, not automatic | operator writes; `gcserv` re-aligns ciphertext | None: latest two versions cached |
| `volcreds` (per volume) | Never (write-once, immutable) | dataserv/gcserv | n/a |
| `s3creds` / DB creds | Your policy | operator | Per backend |

### Service key rotation

Generate new keypairs, write each service's own secret and the matching
`service-verifiers` entry in the Hub vault, replicate verifiers to all Region
vaults. Services keep the last two keys cached, so the old key stays valid through
the next cycle and no restart is needed. Ed25519 keys are raw bytes, standard
base64 (signing 88 chars, verification 44 chars); wrapped PEM/DER/PKCS8 is
rejected.

### api-master rotation (per region)

Write a new `api-master` version with the backend's native mechanism. The
user-facing API key string is unchanged; only the stored ciphertext is
re-encrypted. `gcserv`'s alignment goal picks up the new version and re-encrypts
stored API-key ciphertexts; the latest two versions are kept for a decryption
grace window.

```bash
# HashiCorp; one field 'key' only (>=12 chars). A second field breaks rotation.
vault kv put mountos/api-master 'key=<value>'
```

### Refresh caches after any rotation

Call the single vault operation to refresh appserv's verifier cache plus the
master-key cache, then it broadcasts to the other services:

```ts
await client.vault.resync()   // POST /api/v1/vault/resync, rate-limited 2/min, system-admin scope
```

`volcreds` are write-once: dataserv/gcserv refuse to overwrite a volume key once
the volume holds data, because re-keying would require re-encrypting all data. A
volume's `volcreds` is deleted only when the volume is removed (see section 6).

## 5. Discovery and health

The HUB is the discovery hub backed by the admin DB. Watch what registered and
what is failing.

```ts
const nodes = await client.serviceNodes.list(regionId)   // optionally filter by cluster/type/status
```

| Read | REST | Use |
| --- | --- | --- |
| Region nodes | `GET /regions/:regionId/nodes?serviceType=&status=&regionClusterId=` | Confirm a pool scaled; spot stale heartbeats (`lastHeartbeat`, `status`) |
| Node stats | `GET /regions/:regionId/nodes/:nodeId/stats` | Per-node detail |
| All nodes | `GET /nodes?serviceType=&status=&inactiveHours=` | Deployment-wide view |
| Region alerts | `GET /regions/:regionId/alerts/list?severity=&category=&regionClusterId=&active=true` | Topology and runtime alerts |
| Alert count | `GET /regions/:regionId/alerts/count?regionClusterId=` | Dashboards |
| Resolve alert | `POST /regions/:regionId/alerts/:alertId/resolve` | Clear handled alerts |
| Client sessions | `GET /client-sessions/list?regionId=&regionClusterId=&volumeId=` | Who is mounted, on which cluster/fork |
| Region audit | `GET /regions/:regionId/audit-logs/list?subject=&node=` | Trace moves, registrations, deactivations |

Two failure signatures to recognize:

- **Strict-registration failure.** A service started with an empty or unknown
  `REGION_CLUSTER_ID`, or one referencing a deactivated cluster, refuses to
  register and raises a critical topology alert. The node will not appear in
  `/nodes`. Fix the env; do not retry blindly.
- **Not-ready cluster.** A cluster with `isReady=false` accepts no volumes and the
  HUB will not route to it. If a new cluster never went ready, no cluster-scoped
  service registered into it. Bring up `dataserv` there, or force it with
  `set-ready { "ready": true }` only after confirming a serving pool exists.

## 6. Volume lifecycle

| State change | REST | Effect |
| --- | --- | --- |
| Lock | `POST /volumes/:volumeId/lock` | Read-only freeze; mounts stay up, writes rejected |
| Unlock | `POST /volumes/:volumeId/unlock` | Resume writes |
| Deactivate | `POST /volumes/:volumeId/deactivate` `{ isCleanupMetaEnabled?, isCleanupStorageEnabled?, isCleanupVaultEnabled? }` | Take the volume out of service; flags arm staged cleanup |
| Activate | `POST /volumes/:volumeId/activate` | Bring a deactivated volume back |

Lock first when you need a stable point (before a move under heavy write load, or
before deactivation) without unmounting clients.

Deactivation cleanup flags arm staged reclamation run by `gcserv`, gated by the
volume's grace period and ordered meta then storage then vault:

| Flag | Reclaims |
| --- | --- |
| `isCleanupMetaEnabled` | Metadata (directory tree, forks, versions) |
| `isCleanupStorageEnabled` | Object-store segments/blobs |
| `isCleanupVaultEnabled` | `volcreds` entry, only after meta and storage cleanup completed, grace passed, and the volume actually used encryption |

Leave the flags unset to deactivate without destroying data, so `activate` can
restore the volume. Setting cleanup flags is destructive and not reversible by
`activate` once reclamation runs.

## 7. Live dashboards

Every service and the client have a built-in dashboard for live observability, bound to the
running process over its control socket. No external monitoring stack is required for a quick look.

- Service: `<service> dashboard [pid]` (for example `dataserv dashboard`). Defaults to a terminal
  UI; pass `--gui` for a local browser GUI. Shows the node's health, throughput, Raft state
  (dataserv), cache occupancy, and live sessions.
- Client: `mountos dashboard [mount-path]`, same TUI-by-default / `--gui` split, for a mount.

For configuration, each binary prints its own env template with `<service> env` (or
`<service> env -w .env`), the authoritative, build-specific list.

For fleet-wide, longer-retention observability (internal ops use, not customer-facing),
`monitoring/` has a self-contained VictoriaMetrics + Grafana + vmalert + Alertmanager
docker-compose stack. Target discovery and every `/metrics` scrape are gated by a metrics
token minted from appserv (`POST /api/v1/metrics/token`, admin-only, see the Observability
section of `04-configuration.md`); copy `monitoring/.env.sample` to `monitoring/.env` with
that token as `METRICS_AUTH_TOKEN`, then `docker compose -f monitoring/docker-compose.yml up`.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| New pool node never appears in `/nodes`, critical topology alert | Empty, unknown, or deactivated `REGION_CLUSTER_ID` | Correct the env UUID; it must reference an active `region_clusters` row. Restart |
| Discovery rejects a `regionClusterId` on a request | Cluster belongs to a different region than the volume | Pass a cluster in the volume's own region, or omit it and let the HUB resolve |
| Volume create / move rejected: cluster not accepting | Target cluster `isReady=false` or `isActive=false` | Register a cluster-scoped service into it (auto-readies), or `set-ready`; reactivate if deactivated |
| Service starts but refuses to register on a cluster (topology alert, retries every minute) | Cluster is deactivated | Use an active cluster, or reactivate it; deactivated clusters reject registration by design |
| Clients still hit the old cluster after a move | Inside the handover window | Expected until `handoverUntil`; source keeps serving and hints. Clients re-resolve after the deadline |
| `vault.resync` returns rate-limited | More than 2 calls/min | Wait; the broadcast already refreshes all services |

## Read next

- Index: https://mountos.io/skill.md
- Provision the HUB, accounts, regions, and region services: https://mountos.io/skills/provision.md
- Create volumes and access keys: https://mountos.io/skills/volumes.md
- Integrate S3, WebHDFS, and CSI: https://mountos.io/skills/integrate.md
- Full topics: https://mountos.io/ai/topics/architecture.md, https://mountos.io/ai/topics/deployment.md, https://mountos.io/ai/topics/secrets.md, https://mountos.io/ai/topics/configuration.md
- Symptom-to-fix reference with verbatim error strings: https://mountos.io/ai/topics/troubleshooting.md
- Admin API reference: https://github.com/mountos-io/mountos-admin-sdk (`api.md`)
- Support: support@mountos.io