# mountOS Skill: S3 clients and multipart tuning

> Reach a mountOS volume over the S3 API from any tool or SDK, and configure multipart uploads so large writes are accepted. One rule matters across every client (the multipart part-size floor); most clients already satisfy it by default.

This skill assumes a general (object) volume with an access-key pair already exists. To stand
up the HUB, a region, a storage, or a volume first, see https://mountos.io/skills/provision.md and
https://mountos.io/skills/volumes.md. For the broader integration picture (WebHDFS, CSI, the Admin
SDK), see https://mountos.io/skills/integrate.md.

## S3 surfaces

mountOS exposes its S3 REST surface from the `mountos` client, authenticated with **AWS
SigV4-compatible** signing using service name `s3` and the volume access key pair (`apiKey` /
`apiSecret`). There is no separate gateway service to deploy. `blockserv` is the block-storage
byte backend, reached over the block protocol, not an S3 surface.

| Surface | Command | Use when |
| --- | --- | --- |
| S3 REST, no mount | `mountos gateway --gateway s3 -a <key> -s` | Any S3 SDK or tool that needs buckets, listings, multipart, lifecycle |
| S3 REST alongside a mount | `mountos mount -m <dir> --gateway s3 ...` | The same host wants both a filesystem view and an S3 endpoint |

The listener binds `127.0.0.1` on an auto-assigned port. Pin it with `--gateway-port s3=9000`,
and use `--gateway-no-loopback` with `--gateway-cert` / `--gateway-key` to serve other hosts
over TLS. Each running gateway writes its endpoint URLs to a descriptor JSON under
`~/.mountOS/gateway/`.

General S3 semantics are standard SigV4 with a per-request list cap of 1000 keys and the
multipart rule below. The gateway serves general object volumes only. (The iceberg lake plane
has its own S3 endpoint, covered at the end.)

## The multipart rule (the one thing to get right)

For a multipart upload, every part **except the last** must be at least **8 MiB**
(`8388608` bytes). The last part may be any size. A non-last part below 8 MiB is rejected with
`EntityTooSmall`. The part count is capped at 10,000, the standard S3 limit.

An 8-MiB-**multiple** part size (8, 16, 24 MiB, ...) is **recommended** for read performance
(it aligns to the storage block grid, so each block is one GET), but it is **not required**.
Any part size at or above the 8 MiB floor is accepted; off-grid sizes read correctly and only
pay a small partial-block fetch at part boundaries.

So the only thing a client must avoid is a non-last part smaller than 8 MiB.

## Client matrix

Most clients pick a default part size at or above 8 MiB, so they work with no change. The one
common exception is rclone.

| Client | Default non-last part size | Works as-is | If you need to change it |
| --- | --- | --- | --- |
| AWS CLI (`aws s3`) | 8 MiB, doubles as the object grows | yes | `aws configure set default.s3.multipart_chunksize 8MB` |
| boto3 / AWS SDKs | 8 MiB, doubles as the object grows | yes | `TransferConfig(multipart_chunksize=8*1024*1024)` |
| Cyberduck / Mountain Duck | 10 MiB | yes | Preferences, S3, default upload chunk size |
| rclone (`s3` backend) | **5 MiB (below the floor, rejected)** | **no** | `--s3-chunk-size 8Mi` on the command, or `chunk_size = 8Mi` in the remote config |
| DuckDB (`httpfs`) | derived, about 80 MB by default | yes | see DuckDB note below |
| Trino (native S3) | 32 MB | yes | `s3.streaming.part-size = 8MB` (range 5MB to 256MB) |
| PyIceberg (PyArrow FileIO) | managed by PyArrow / AWS SDK | yes | see PyIceberg note below |

### rclone

rclone's default chunk size on the `s3` backend is 5 MiB, which is under the floor and gets an
`EntityTooSmall`. Raise it to 8 MiB or more.

```
rclone copy ./data mos:bucket/ --s3-chunk-size 8Mi
```

Or set it once in the remote config (`rclone config`), key `chunk_size = 8Mi`.

### DuckDB

DuckDB's `httpfs` extension does not take a part size directly. It derives one as
`s3_uploader_max_filesize / s3_uploader_max_parts_per_file`. The defaults (about 800 GB over
10,000 parts) give a part size near 80 MB, well above the floor, so no change is needed. To
tune it, set those two options (for example a smaller `s3_uploader_max_filesize` raises the
per-part count and lowers the part size). There is no `s3_uploader_max_parts_size` or
`s3_uploader_background_threads` option; those names do not exist.

### Trino

Use the native S3 file system (`fs.native-s3.enabled=true` on current releases). The part size
is `s3.streaming.part-size` (default 32 MB, range 5 MB to 256 MB), already above the floor. The
keys `fs.native-s3.multipart.part-size` and the legacy `hive.s3.*` Hadoop S3A settings are not
used; the legacy S3 file system was removed from Trino.

### PyIceberg

PyIceberg's default PyArrow FileIO manages multipart internally through PyArrow and the AWS
SDK; it does not honor `s3.multipart.part-size-bytes` (that is a Java-Iceberg S3FileIO
property). Small table writes are a single PUT, so they are unaffected. Large writes use
PyArrow's defaults, which sit at or above the floor.

## Iceberg lake S3 endpoint

Iceberg (lake) volumes are not served by the general S3 gateway. They are reached with the
`mountos` client running its lake endpoints (an Iceberg REST catalog and a separate
lake S3 listener), which query engines (DuckDB, Trino, Spark, PyIceberg) point at directly. The
same 8 MiB multipart floor applies on that endpoint. Engines discover the lake S3 endpoint from
the catalog's `/v1/config` response, so they need no separate S3 endpoint setting.

## Read next

- Integrate over S3, WebHDFS, CSI, and the Admin SDK: https://mountos.io/skills/integrate.md
- Create and use volumes and keys: https://mountos.io/skills/volumes.md
- Topic index: https://mountos.io/llms.txt
- Support: support@mountos.io and https://mountos.io/support