Set up a region

A region stores data. It has its own database and its own vault. All of the region's metadata clusters share that database and vault. The region's storage records point at one or more S3-compatible or Azure stores. The region writes data to those stores. A metadata cluster is a volume-load partition. It owns a set of volumes and serves them.

Take the region and metadata cluster IDs

You created the region during the HUB setup. That step also created its default metadata cluster, uno. Use uno, or add a metadata cluster when one partition is not enough.

// the region from the HUB setup already has its default metadata cluster, "uno"
// add a metadata cluster only to partition volume load further
const { id: clusterId } = await client.regionClusters.create(regionId, { name: 'dos' })

// the region from the HUB setup already has its default metadata cluster, "uno"
// add a metadata cluster only to partition volume load further
cluster, err := client.RegionClusters.Create(ctx, regionID, &sdk.CreateRegionClusterRequest{Name: "dos"})

// the region from the HUB setup already has its default metadata cluster, "uno"
// add a metadata cluster only to partition volume load further
let cluster = client.region_clusters.create(region_id, &CreateRegionClusterRequest {
    name: "dos".into(),
}).await?;

Every region and metadata cluster record carries an exportId. The region's services take the metadata cluster's value as REGION_CLUSTER_ID. They resolve it against the HUB at startup. No separate REGION_ID exists.

Provision the region's infrastructure

Outside the Admin API, give the region its own database and a Region vault. Copy the verifier set from the Hub vault into the Region vault. mountOS does not replicate vaults automatically. One database and one vault serve the whole region. A new metadata cluster needs neither.

A storage record points at existing object storage. It does not provision that storage. The record holds an endpoint, credentials, and some configuration for any S3-compatible or Azure store, S3 and R2 included. Put the store in the same locality as the region for best performance. The store does not need to be self-operated. To use block devices as the backend, run a block storage.

Block storage

A block storage gives a volume's content a region-local home on raw block devices, with no primary among its servers. Object storage sits behind it as the durable source of truth. Clients reach a volume's storage nodes through HUB discovery, not DNS.

Each blockserv server runs with its own BLOCK_VOLUME_ID and REGION_CLUSTER_ID. It caches object-storage parts on its block device. The public surface is the block protocol, not an S3 endpoint. An admin can drain a server for maintenance; the volume picks up a replacement without downtime.

The Region vault

Each service reads its secrets from the Region vault under its own key, mountos/dataserv, mountos/gcserv, and so on. Seed each key with these values.

• DB_URL is the regional database connection.
• DB_DIALECT picks the schema, postgresql or mysql.
• DB_PROVIDER_VERSION is the engine version for capability detection, like 16 or 8.0.32.
• ED25519_SIGNING_KEY is the service's own private signing key.
• ED25519_VERIFICATION_KEY is its matching public key.

The database values matter only to dataserv and gcserv. Only those two services touch the regional database. Every service carries its own key pair. The HUB checks a joining service against the verifier set. That is why you copy the set in from the Hub vault.

Every service machine needs a public IPv4 and a private one. No regional service needs DNS. The HUB's discovery reaches dataserv, gcserv, and blockserv by address, not by DNS. Discovery also hands clients those addresses. gcserv stays hidden inside the region. You reach gcserv only through the HUB.

Two addresses per node

Each service advertises both of its addresses, and the two have different jobs. Clients reach a node on its public address. Peer connections and the HUB-to-region RPC use the private one. Registration fails outright when no private address resolves, and the peer port is private-only with no public fallback.

Do not set ADVERTISE_ADDR on a machine that has both. Supplying it forces explicit-address mode, which mirrors that one address into both roles. Pin it to the public IP and every peer tries to reach that public address from inside your own network. Most clouds do not route an instance's public address back to a machine in the same virtual network, so the failure is a silent timeout with nothing in the log naming the cause. Leave it unset and the service detects both addresses from instance metadata. Set it only on a machine whose sole reachable address is private.

Open both peer ports between region services. The peer replication port runs on APP_PORT+1, but a joining node dials an existing peer's RPC port to ask for admission. With only the replication port open, every node loops on no peer accepted join request and looks healthy on its own while the cluster never actually forms.

When gcserv runs on the same machines as dataserv, give it its own PORT and pin its RPC_PORT explicitly. The two share one environment file and both default to the same HTTP port, so one of them crash-loops while the node still reports healthy. Pinning matters because RPC_PORT derives from PORT, so moving PORT alone silently moves the RPC port out from under your firewall rules.

Size the dataserv nodes

dataserv is the metadata backbone. Each metadata cluster runs three instances. Every metadata operation from every mount flows through these instances. Give these machines large RAM and high network bandwidth.

dataserv serves hot metadata from memory. METAENGINE_ARENA_SIZE sets the metadata cache size. The minimum is 128MB. Size the cache to the metadata working set and size the machine's RAM around it. Confirm the value the service adopted in its startup log. Keep RAFT_PORT and RAFT_DATA_DIR stable across restarts.

A cold fleet takes minutes to reach quorum, and that is normal. Records for terminated nodes linger, and a fresh node waits for those phantom peers to age out of the participant set before it bootstraps. Measured on a three-node region: nodes healthy in about 90 seconds, full quorum at about six minutes. Do not read the join messages during that window as a deadlock, and do not start deleting node records. Wait at least ten minutes before diagnosing.

Bring up the services

Each service follows the install pattern. Its env -w .env command writes the template. The template documents every variable inline. The environment file carries these values.

• SERVICE_RPC_ADDR is the HUB address the service registers and heartbeats against.
• REGION_CLUSTER_ID is the metadata cluster export ID above. It carries the region. No separate REGION_ID exists.
• VAULT_PROVIDER names the secret store and carries its address and role credentials.
• METAENGINE_ARENA_SIZE sets the dataserv metadata cache size.

Region services need no license file. The license arrives from the HUB over the heartbeat. MOUNTOS_LICENSE_PATH is an appserv-only convenience for seeding the first license, and region services ignore it.

The regional schema belongs to the database pair, dataserv and gcserv. Run db install once on the fresh regional database. It reads DB_URL and DB_DIALECT from the Region vault, downloads the schema for that dialect, and applies it in the same step. The service caches the schema locally and reuses it on later runs.

db install is not idempotent. Once the schema exists it exits non-zero with an already-installed error, on a restart, on a replacement machine, and on the second node that shares the database. If you write your own unit file, make it best-effort rather than a hard start pre-condition. Under systemd, prefix the ExecStartPre= line with -. Without that, the service starts once and is blocked forever afterwards, and the failure appears on the second start rather than the first.

./mountos-install --pkg mountos-dataserv
mountos-dataserv env -w .env     # the template documents every variable inline
export MOUNTOS_LICENSE_PATH=$PWD/license
set -a
. ./.env

mountos-dataserv db install      # create the regional schema, once per region
mountos-dataserv                 # start, register with the HUB, heartbeat

To upgrade an existing region later, run db migrate. It applies only the new migrations in range for the binary. The command is idempotent. It is safe to run from multiple instances at the same time.

mountos-dataserv db migrate   # apply pending schema changes, then restart

On boot a service registers itself with the HUB and heartbeats. The first registration into uno flips it to ready. The HUB can then assign volumes to it. A service can point at a metadata cluster in the wrong region, or at a deactivated metadata cluster, by mistake. That service refuses to register. It raises a topology alert. It retries until you fix the topology.

S3 and WebHDFS are protocol surfaces of the mountos client. Each surface binds to the one volume the client authenticates for. There is no multi-tenant gateway fleet to deploy. A region needs no extra provisioning for these surfaces.

Verify

The dashboard's region detail shows each metadata cluster with its nodes, health, and the region audit log. The nodes view shows every registered instance with memory, load, and heartbeat. See Metadata clusters and Regional internals for how the pieces serve a mount.