Skip to content

Architecture

Regional internals

A region belongs to one account. It has its own regional database and region vault, and file content lives in the S3-compatible or Azure stores its storages point at. A region subdivides into one or more region clusters, volume-load pools that share the region's single database and vault. Cluster-scoped services, dataserv and gcserv, each serve only their cluster's shard of volumes. A volume lives in exactly one region, and its content never crosses a region boundary at serving time.

Inside one region
Region: eu-west-1regional data planepartitioned into one or more region clusters (shared DB + vault)INGRESS (regional DNS)dataserv.regionmetadatablock storageblock protocoldataserv · raft cluster3 or 5 nodesnode-0leaderarena cachenode-1followerarena cachenode-2followerarena cacheraft replication + ownership mapblock storageone of manyblock volume Acluster Ablock volume Bcluster Bblock volume Ccluster Cactive-active, distinct clustersdiscovered via appservgcserv poolgarbage collection · retention cleanupRegion DBMySQL / PostgreSQLStorage backendsS3-compatible or Azure · one or many · via storagesGlobal appserv (control plane)discovery · auth · adminSQLcleanuppersist / fetch partsheartbeatregional trafficraft / heartbeatgcserv can run on the same machines as the dataserv nodes

dataserv

dataserv is the metadata backbone. It holds the directory tree, files, forks, and versions. It runs as an odd-sized raft cluster, three or five nodes, so a node can fail without losing writes.

Each node owns a subset of the volume's partitions and keeps a hot working set in memory, backed by the regional database. A request can land on any node and is served by the owning node.

Each dataserv pool is cluster-scoped. It serves only the volumes pinned to its region cluster. Multiple clusters can run side by side in one region, sharing that region's single database and vault.

gcserv

gcserv is the housekeeping process, run as its own pool per cluster and scaled independently of the serving path. It collects orphaned blocks, prunes old forks past the retention floor, re-wraps API-key secrets when the region vault rotates, and clears volume secrets after a volume is deactivated.

blockserv

A block storage is a peer mesh of up to three active-active block volumes. Each block volume pairs one blockserv node with its own attached volume, and the block volumes sit in distinct region clusters for fault isolation. The block volumes are peer-to-peer with no primary, replica, or failover hierarchy, and they replicate bidirectionally. blockserv exposes the block protocol as its public API. It has no public S3 surface and no SigV4. Where it runs, it validates each request and translates it directly into reads or writes against its block volume. The bytes never go through dataserv.

Object storage is the durable source of truth. Each block volume caches parts on a raw block device and stays current with the durable store, so blockserv is a caching tier rather than the durable backend.

Block volumes span distinct clusters by design. Clients discover the live block volumes through appserv, the discovery endpoint, not through DNS, so blockserv needs no DNS entry. The client data plane listens on BLOCK_PORT, and peer replication runs on a neighboring port. Clients reconnect across block volumes without explicit failover.

Nothing to deploy for S3 or WebHDFS

S3 and WebHDFS are protocol surfaces of the mountos client, bound to the one volume the client is authenticated for. There is no multi-tenant gateway fleet to deploy. A region serves those callers through the same dataserv and blockserv paths a mount uses, so sizing a region never involves a gateway tier. See Clients.

Backing storage

  • The regional database is MySQL or PostgreSQL wire-compatible. It holds the durable file tree, forks, blocks, quotas, and session state. dataserv serves the working set from memory, keeping load off the database.
  • Storage backends are the S3-compatible or Azure stores the region's storages point at, one or many per region. AWS S3, Google Cloud Storage, Backblaze B2, MinIO, or Azure Blob Storage. Object storage is the durable source of truth where all file content lives. blockserv is not a durable backend: each block volume caches parts on a raw block device.
  • The Region vault holds region-scoped secrets for every regional service, shared across all clusters in the region. See Vault and handshake.

Sizing a region

  • dataserv runs an odd count, three for most deployments, five to tolerate two concurrent node losses. One cluster per region by default. Add clusters to spread volume load.
  • gcserv runs as its own pool per cluster, sized to the background workload.
  • blockserv forms a block storage of up to three active-active block volumes. Each block volume is one blockserv node plus its attached volume, placed in a distinct cluster for fault isolation, and the block volumes replicate peer-to-peer with no primary or failover.
  • The regional database is whatever the ops team already runs, one per region.
  • Storage backends are any S3-compatible or Azure stores, one or many.
esc
↑↓ to navigate ↵ to open