Skip to content

Architecture

Regional internals

A region belongs to one account. The region has its own regional database and region vault. Its storages point at S3-compatible or Azure stores. Those stores hold the file content. A region subdivides into one or more metadata clusters, volume-load pools that share the region's single database and vault. Cluster-scoped metadata nodes each serve only their metadata cluster's shard of volumes. A volume belongs to exactly one region. Its content never crosses a region boundary at serving time.

Inside one region
Region: eu-west-1regional data planepartitioned into one or more metadata clusters (shared DB + vault)CLIENTS (direct IP from HUB discovery, no regional DNS)metadatametadatablock storageblock protocolmetadata · metadata cluster3 nodesnode-0nodecachenode-1nodecachenode-2nodecachereplicated for fault toleranceblock storageone of manypair 1server Aserver Bpair 2server Aserver Bno primaryeach volume uses its own pairsdiscovered via the hubgc poolgarbage collection · retention cleanupRegion DBMySQL / PostgreSQLStorage backendsS3-compatible or Azure · one or many · via storageshub (control plane)discovery · auth · adminSQLcleanupSQLpersist / fetch partsheartbeatregional trafficreplication / heartbeatgc runs on the same machines as the metadata nodes

Metadata nodes

The metadata node is the metadata backbone. It holds the directory tree, files, forks, and versions. Each metadata cluster runs three metadata nodes.

The pool distributes metadata work across the nodes in its metadata cluster and keeps volume data available as nodes are maintained.

Each metadata-node pool is cluster-scoped. It serves only the volumes that belong to its metadata cluster. Multiple metadata clusters can run side by side in one region. They share that region's single database and vault.

The metadata node also runs the background work for its metadata cluster. It collects orphaned blocks. It prunes old forks past the retention floor. It re-wraps API-key secrets when the region vault rotates. It clears volume secrets after a volume deactivates.

Storage nodes

Storage nodes are optional block storage, with no primary among them. They hold file content on local block devices and sync durable writes to object storage, the source of truth. Their public API is the block protocol, with no S3 surface and no SigV4. Reads and writes never go through the metadata node.

Clients discover a volume's storage nodes through the HUB, the discovery endpoint, not through DNS. The client data plane listens on BLOCK_PORT. An admin can retire a server; the volumes that used it pick up a replacement without downtime.

Nothing to deploy for S3 or WebHDFS

S3 and WebHDFS are protocol surfaces of the mountos client. Each surface binds to the one volume the client authenticates for. There is no multi-tenant gateway fleet to deploy. A region serves those callers through the same metadata-node and storage-node paths a mount uses. Sizing a region therefore never involves a gateway tier. See Clients.

Backing storage

  • The regional database is MySQL or PostgreSQL wire-compatible. It holds the durable file tree, forks, blocks, quotas, and session state. The metadata node serves the working set from memory. This reduces load on the database.
  • Storage backends are the S3-compatible or Azure stores the region's storages point at, one or many per region. Examples include AWS S3, Google Cloud Storage, Backblaze B2, MinIO, and Azure Blob Storage. Object storage is the durable source of truth. It holds all file content. Each storage node holds parts on a raw block device and syncs durable writes to object storage.
  • The Region vault holds region-scoped secrets for every regional service. All metadata clusters in the region share this vault. See Vault and handshake.

Sizing a region

  • Metadata nodes run three to a metadata cluster. One metadata cluster serves the region by default. Add metadata clusters to spread volume load.
  • Storage nodes are optional block storage, with no primary among them. Size the node count for the volumes' capacity and throughput needs, plus headroom for maintenance.
  • The regional database is whatever the ops team already runs, one per region.
  • Storage backends are any S3-compatible or Azure stores, one or many.
esc
↑↓ to navigate ↵ to open