s3dol

s3dol — S3 and S3-compatible object storage behind a dict-like interface.

The one-liner:

import s3dol
s = s3dol.s3_store('my-bucket')          # MutableMapping[str, bytes]
s['k'] = b'v'; s['k']; list(s); del s['k']

Keyed capabilities are sibling stores you index (never methods — a dol key wrapper would hand a method the unmapped key; ADR-0011):

s3dol.handles(s)['k']    # ObjectHandle (ranged reads, streams, metadata)
s3dol.urls(s)['k']       # presigned URL
s3dol.info(s)['k']       # ObjectInfo

Non-keyed operations are free functions taking the store first:

s3dol.sub(s, 'folder/')          s3dol.prefixes(s)
s3dol.delete_many(s, keys)       s3dol.delete_bucket(endpoint, name, force=True)

Test without a cloud: from s3dol.testing import mock_s3, run_conformance. Migrating from v0? s3dol.diagnose(**your_S3Store_kwargs) prints what resolves, from where, and whether v1 moves it. The legacy S3Store keeps working via s3dol.store (deprecated, removed in v2).

Everything here loads lazily (PEP 562): import s3dol does not import boto3.

exception s3dol.AccessDenied

Permission denied. Not a KeyError — a missing permission must not look like a missing key (ADR-0004 §1/§3).

class s3dol.BucketCollection(bucket: str, *, connection: S3Connection | None = None, prefix: str = '', delimiter: str = '/', reads: ReadStrategy | None = None, writes: WriteStrategy | None = None, on_missing_bucket: str = 'assume', strict_delete: bool = False)

Keys of one bucket (under the prefix). Collection only — iteration and membership, no reads.

class s3dol.BucketHandles(bucket: str, *, connection: S3Connection | None = None, prefix: str = '', delimiter: str = '/', reads: ReadStrategy | None = None, writes: WriteStrategy | None = None, on_missing_bucket: str = 'assume', strict_delete: bool = False)

k -> ObjectHandle (key bound at construction; zero round-trips).

class s3dol.BucketInfo(bucket: str, *, connection: S3Connection | None = None, prefix: str = '', delimiter: str = '/', reads: ReadStrategy | None = None, writes: WriteStrategy | None = None, on_missing_bucket: str = 'assume', strict_delete: bool = False)

k -> ObjectInfo (one HeadObject).

exception s3dol.BucketNotEmpty

Refusal to delete a non-empty bucket. Use s3dol.delete_bucket(endpoint, name, force=True) for the explicit cascading form (ADR-0010 §3).

exception s3dol.BucketNotFound

The bucket is absent.

Still a KeyError: for an object operation it is a key-space problem for the caller; for a bucket operation it is the key of the endpoint mapping.

class s3dol.BucketReader(bucket: str, *, connection: S3Connection | None = None, prefix: str = '', delimiter: str = '/', reads: ReadStrategy | None = None, writes: WriteStrategy | None = None, on_missing_bucket: str = 'assume', strict_delete: bool = False)

• __getitem__ -> bytes (or the read strategy’s value type).

url_for(k: str, *, expires_in: int = 3600, client_method: str = 'get_object', **params) → str | None

Presigned URL for k — the one keyed method (ADR-0011 D3b), kept solely because dol.SupportsUrlFor requires a method and dol.content_url reaches it by getattr. The canonical form is s3dol.urls(store)[k].

Guarded to be correct-or-loud, never silently wrong:

• unwrapped store: k is a key in this store’s own key space, so the URL addresses self._id_of_key(k). Correct, and this is also the contract dol.content_url calls under — it resolves the key through the outer layers and stops at the layer owning url_for, “because that layer applies its own” (dol/content.py);

• this leaf has ever been wrapped: raises. Two call paths then reach this one signature with different key domains and nothing distinguishes them: dol.content_url hands over a key already mapped through the wrapper layers, while a direct wrapped.url_for(k) (delegation) hands over the raw outer key. Applying _id_of_key is right for the first and wrong for the second; re-resolving through the chain is right for the second and double-transforms the first. Since a wrong presigned URL addresses a different real object, the only honest answer is refusal, naming the sibling store — which is correct by construction because it goes through __getitem__ (ADR-0011 D2/D3b, and see the note below);

• anonymous connection: returns None (SupportsUrlFor is Optional[str]; raising would turn a public-bucket streaming fallback into a 500 — ADR-0012 D5). The check reads the built client’s signature version, never spec.anon.

Note

This refusal is a finding, not a limitation we chose. ADR-0011 D3b expected a guarded method to serve both paths; it cannot, on a leaf that owns its prefix. Reported upstream on dol#86/#83 as evidence that a keyed capability method is unfixable in the general case, and that the sibling-store form is the only one correct by construction.

class s3dol.BucketStore(bucket: str, *, connection: S3Connection | None = None, prefix: str = '', delimiter: str = '/', reads: ReadStrategy | None = None, writes: WriteStrategy | None = None, on_missing_bucket: str = 'assume', strict_delete: bool = False)

• __setitem__ / __delitem__. The user-facing read-write store.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

class s3dol.BucketUrls(*args, expires_in: int = 3600, client_method: str = 'get_object', **kwargs)

k -> presigned URL (zero object requests; None when anonymous). The canonical spelling of what url_for shims (ADR-0011 D3b).

class s3dol.CallableCredentials(fetch: Callable[[], Mapping])

A picklable zero-arg callable returning a mapping with keys access_key/secret_key (aliases: key/secret), optional token, optional expires_at/expiry_time (ISO 8601). With an expiry, credentials are refreshable — the callable is re-invoked by botocore’s refresh machinery (advisory 900 s / mandatory 600 s before expiry).

provenance() → str

A label for diagnose() — never a credential value.

class s3dol.Capabilities(list_objects_v2: bool = True, batch_delete: bool = True, presigned_post: bool = True, multipart: bool = True, max_multipart_parts: int = 10000, min_part_size: int = 5242880, object_tagging: bool = True, versioning: bool = True, conditional_writes: bool = True, consistency: Literal['strong', 'read-after-write', 'eventual'] = 'strong')

What a backend supports. A static declared table, never a probe (a probe costs a round-trip, is wrong under partial permissions, and can’t be trusted anyway — ADR-0003 §2). Missing capabilities are handled per-capability: emulate when exact (batch delete -> loop), substitute when equivalent (ListObjectsV2 -> V1), raise NotSupported when there is no honest fallback. Never silently degrade correctness.

class s3dol.Chunks(chunks: ~typing.Iterable[bytes], consumed: list = <factory>)

The value is the concatenation of an iterable of bytes chunks. One-shot unless the iterable is re-iterable (a list is; a generator is not — and a consumed generator writes b'', which is why assigning the same one-shot ref twice is a documented error).

exception s3dol.ConfigurationError

A construction- or resolution-time misconfiguration (ADR-0012 D1).

exception s3dol.CredentialsError

Credentials are missing, expired, or invalid.

class s3dol.EndpointCollection(connection: S3Connection | None = None, **bucket_kwargs)

Bucket names at one endpoint.

class s3dol.EndpointReader(connection: S3Connection | None = None, **bucket_kwargs)

• __getitem__ -> BucketReader.

class s3dol.EndpointStore(connection: S3Connection | None = None, **bucket_kwargs)

• bucket creation/deletion. del endpoint[name] refuses a non-empty

bucket; the explicit cascading form is the free function s3dol.delete_bucket(endpoint, name, force=True) (ADR-0010 §3 as amended by ADR-0011 D4 — a public keyed destructive method is the shape the family census found destroying wrong data).

class s3dol.Filepath(path: str)

The value is the content of this file. Re-readable (N1-total).

exception s3dol.KeyNotValid

The key is syntactically invalid (checked before the wire).

Inherits from both KeyError and ValueError deliberately, so either except works. Named KeyNotValid (not KeyValidationError) because dol already ships two different classes called KeyValidationError (ADR-0004 §5).

exception s3dol.NotSupported

The operation is not supported by this provider (names both).

exception s3dol.ObjectArchived(message: str, *, storage_class: str | None = None, restore_status: str | None = None, restore: Callable | None = None)

The object exists but is archived (e.g. Glacier) and cannot be read now.

A KeyError, deliberately and arguably (ADR-0004 §4): k in store stays True, but store[k] fails as a KeyError so that store.get(k, default) degrades to the not-available branch. Note that only the defaulted accessors degrade — dict(store), .items() and Mapping.__eq__ still raise. setdefault/pop are overridden in the store to re-raise this instead of silently overwriting the archived object.

restore(days: int = 1, *, tier: str = 'Standard')

Request restoration of the archived object (if a handle was bound).

class s3dol.ObjectHandle(bucket: str, key: str, *, connection: S3Connection | None = None)

One object, key bound at construction — which is what makes it immune to the delegation trap (a key codec over the store cannot corrupt a key that was resolved before the handle existed; ADR-0011 D1, following azuredol.BlobHandle). This is where ranged reads, streaming, download-to-disk, metadata and presigned URLs live.

download_to(path) → str

Download to disk (never fully in memory). Accepts a path or Filepath; returns the path written.

info() → ObjectInfo

One HeadObject -> ObjectInfo.

read(*, offset: int | None = None, length: int | None = None) → bytes

The object’s bytes; a ranged read when offset/length given.

restore(days: int = 1, *, tier: str = 'Standard') → None

Request restoration of an archived (Glacier) object.

stream(*, chunk_size: int = 8388608) → Iterator[bytes]

Lazy chunk iterator over the body.

url(*, expires_in: int = 3600, client_method: str = 'get_object', **params) → str | None

Presigned URL. Always SigV4; None when anonymous; capped at SigV4’s 604800 s; warns when the signing credential expires sooner (the URL cannot outlive it).

write(value) → None

Write the object (same value domain as store[k] = v).

class s3dol.ObjectInfo(key: str, size: int | None = None, last_modified: datetime | None = None, etag: str | None = None, content_type: str | None = None, storage_class: str | None = None, restore_status: str | None = None, version_id: str | None = None)

Object metadata — from HeadObject or from a LIST row (which already carries size/mtime/etag/storage class that v0 threw away; ADR-0009).

exception s3dol.ObjectNotFound

The object is absent. The Mapping contract’s KeyError.

class s3dol.Preset(name: str, addressing_style: Literal['path', 'virtual'], endpoint_url: str | None = None, region_name: str | None = None, checksum: Literal['when_supported', 'when_required'] | None = None, payload_signing_enabled: bool | None = None, capabilities: Capabilities = Capabilities(list_objects_v2=True, batch_delete=True, presigned_post=True, multipart=True, max_multipart_parts=10000, min_part_size=5242880, object_tagging=True, versioning=True, conditional_writes=True, consistency='strong'), presign_endpoint_url: str | None = None, presign_addressing_style: Literal['path', 'virtual'] | None = None, params: tuple[tuple[str, str], ...] = (), host_patterns: tuple[str, ...] = (), requires_endpoint: bool = False, soft: bool = False, config_kwargs: tuple[tuple[str, Any], ...] = (), error_overrides: tuple[tuple[tuple[str | None, str | None, int | None], str], ...] = (), verified: bool = False)

One provider’s facts. Public, shareable, committable config — which is why there is no credential slot, structurally (ADR-0012 D4).

endpoint_url/region_name may contain {placeholders}; bind them with bind(). A row whose endpoint template has an unbound required placeholder contributes no endpoint rung (ADR-0012 C1).

addressing_style: Literal['path', 'virtual']

Mandatory; no default; 'auto' is illegal (ADR-0012 D4).

bind(**params: str) → Preset

A copy of this row with template params bound.

>>> r2 = get_preset('r2').bind(account_id='abc123')
>>> r2.bound_endpoint()
'https://abc123.r2.cloudflarestorage.com'

bound_endpoint(*, region_name: str | None = None) → str | None

The endpoint URL with params bound, or None if the template has unbound placeholders (C1: an unbound row contributes no rung — preset='minio' silently becoming AWS is structurally impossible).

region_name binds a {region_name} placeholder, per the region ladder’s first rung (ADR-0012 D3).

checksum: Literal['when_supported', 'when_required'] | None = None

None == the row makes no claim (the ladder continues — ADR-0012 D3).

config_kwargs: tuple[tuple[str, Any], ...] = ()

Extra botocore.Config kwargs — allowlisted, scalars only.

error_overrides: tuple[tuple[tuple[str | None, str | None, int | None], str], ...] = ()

Error-classification overrides, consulted before the default table (ADR-0004 §2/§6): rows of ((operation, code, status), kind).

host_patterns: tuple[str, ...] = ()

fnmatch patterns over the endpoint hostname; used for detection (pass 2) and the C2 named-vs-resolved guard.

matches_host(endpoint_url: str) → bool

Whether endpoint_url’s hostname matches this row’s patterns.

missing_endpoint_params(*, region_name: str | None = None) → set

Which placeholders keep bound_endpoint() from producing a URL.

params: tuple[tuple[str, str], ...] = ()

Bound template parameters, e.g. (('account_id', 'abc'),).

property pinned: bool

Whether the row’s region literal is authoritative enough to make a conflicting explicit region_name= raise PresetConflict. Derived, and gated on verified — dormant while rows are doc-sourced (ADR-0012 D4).

presign_endpoint_url: str | None = None

Presigning sometimes genuinely differs from the API config (Hetzner presigns virtual while the API is path; R2 presigns only on the S3 API domain) — ADR-0003 §1.

requires_endpoint: bool = False

The provider cannot be reached without an explicit endpoint.

soft: bool = False

they never win detection over a non-soft row and never trigger the C2 guard.

Type:

Soft rows are last-resort fallbacks (aws, generic-s3)

template_params() → set

All placeholder names across this row’s templates.

verified: bool = False

True only when the row was verified against a live endpoint (with a date in the row’s docstring/comment). Builtin rows are doc-sourced.

class s3dol.ProfileCredentials(profile: str)

Credentials from a named profile — credentials only; the profile’s other config is not selected by this (that is what profile= on the connection is for). Rebuilds live, still-refreshable credentials after unpickling (a frozen dataclass holding only a profile name round-trips).

provenance() → str

A label for diagnose() — never a credential value.

class s3dol.Resolution(endpoint_url: Sourced, region_name: Sourced, signature_version: Sourced, addressing_style: Sourced, checksum: Sourced, payload_signing_enabled: Sourced, verify: Sourced, preset: Preset | None, preset_source: str, credential_provenance: str, capabilities: Capabilities, environ_consulted: tuple[tuple[str, str | None], ...], notes: tuple[tuple[type, str], ...])

The result of resolve() — everything diagnose() prints and everything client-build consumes. Carries provenance labels, never a credential value.

environ_consulted: tuple[tuple[str, str | None], ...]

The non-secret environment keys consulted, with their values.

notes: tuple[tuple[type, str], ...]

Collected (warning class, message) pairs. resolve() never emits them — the impure caller decides (client build warns; diagnose prints rows).

class s3dol.S3Connection(preset: str | Preset | None = None, endpoint_url: str | None = None, region_name: str | None = None, profile: str | None = None, credentials: CredentialProvider | None = None, anon: bool = False, signature_version: str = 's3v4', addressing_style: Literal['path', 'virtual'] | None = None, checksum: Literal['when_supported', 'when_required'] | None = None, payload_signing_enabled: bool | None = None, verify: bool | str | None = None, client_kwargs: tuple = (), deny_means_absent: bool = False)

A frozen, picklable connection spec. No live objects — ever.

Construction performs no I/O and does not import boto3 (ADR-0002); the client is built lazily, once, under a per-instance lock (CPython ≥3.12 removed cached_property’s lock deliberately — measured: 8 racing threads → 8 clients without one).

__post_init__ is the only door: every normalisation and contradiction check happens here, so a bad shape dies at construction with a message, not later inside botocore.

>>> S3Connection(anon=True, profile='prod')
Traceback (most recent call last):
...
s3dol.errors.ConfigurationError: ...

property client

The boto3 S3 client — built lazily, once, under the lock.

property presign_client

The client presigned URLs are generated with.

For most providers this is client. Two providers genuinely need different config for presigning than for the API (ADR-0003 §1): Hetzner presigns virtual-hosted while the API is path-style; R2 presigns only on the S3 API domain. When the resolved preset carries presign_endpoint_url/presign_addressing_style, a second client is built (lazily, once, under the same lock) from the merged config.

resolution(environ: Mapping[str, str] | None=None, aws_config: Mapping | Callable | None = <function load_aws_config>) → Resolution

Resolve against the real environment by default (a thin impure wrapper over the pure resolve()).

s3dol.S3Dol

alias of S3Profiles

exception s3dol.S3DolWarning

Base of every s3dol warning.

exception s3dol.S3Error

Base of every s3dol exception — except S3Error catches the package.

s3dol.S3Jsons

alias of BucketStore

exception s3dol.S3PartialFailure(message: str, *, succeeded: list, failures: dict)

A bulk operation partially failed (ADR-0010 §2).

Not an ExceptionGroup — the package supports Python 3.10.

succeeded

keys the backend reported as deleted (absent keys are reported as deleted too — S3’s semantics, documented, not hidden).

failures

mapping of key -> the error for that key.

s3dol.S3Pickles

alias of BucketStore

class s3dol.S3Profiles(*, bucket_kwargs: dict | None = None, readonly: bool = False)

AWS profile names -> endpoint stores (the v0 S3Dol, renamed — its keys are verifiably profile names, never endpoints; ADR-0007 §1).

>>> # S3Profiles()['prod']  ->  EndpointStore over the 'prod' profile

s3dol.S3Store(bucket_name: str, *, make_bucket: bool | None = None, path: str | None = None, aws_access_key_id: str | None = None, aws_secret_access_key: str | None = None, aws_session_token: str | None = None, endpoint_url: str | None = None, region_name: str | None = None, profile_name: str | None = None, skip_bucket_check: bool | None = None, is_supabase_endpoint: bool | None = None) → MutableMapping

DEPRECATED — the v0 entry point, kept working until v2.

Use s3dol.s3_store() instead: s3_store(bucket, prefix=...).

Signature is v0’s exactly (path= not renamed; make_bucket tri-state maps True -> on_missing_bucket='create', False -> 'raise', None -> 'assume'). Behaviour differences vs v0 are bug fixes only (explicit endpoint/credentials honoured; listing errors raise instead of returning []; writes no longer create buckets unless asked) and announce themselves via S3DolResolutionChanged when they change this call’s resolution.

s3dol.S3Texts

alias of BucketStore

class s3dol.StaticCredentials(access_key: str, secret_key: str, token: str | None = None)

An explicit key pair (optionally a session token).

Note: a static token expires and cannot refresh — prefer profile= or a callable provider for STS/SSO. Pickling this object carries the secret (that is what “explicit credentials” means); its repr does not.

provenance() → str

A label for diagnose() — never a credential value.

class s3dol.Streamable(open_stream: object)

The value is whatever a zero-arg factory’s stream yields — for sources that must be (re)opened at write time, e.g. an HTTP download. The factory is called once per write, so unlike Chunks this ref IS safely re-writable (each write gets a fresh stream).

s3dol.delete_bucket(endpoint, name: str, *, force: bool = False) → None

Delete bucket name. Refuses a non-empty bucket unless force=True, in which case the cascade paginates — v0 listed one page, deleted ≤1000 objects, then failed: a partial, non-idempotent destruction (ADR-0010 §3). A free function, not a method (ADR-0011 D4).

s3dol.delete_many(store, keys: Iterable[str]) → None

Bulk delete (ADR-0010 §2; a free function per ADR-0011 D4 — keyed + destructive + delegated is the census’s data-destroying shape).

Chunks at 1000 (AWS’s cap), parses the Errors list out of the HTTP 200 response, and on partial failure raises one S3PartialFailure carrying .succeeded and .failures. Absent keys are reported by S3 as Deleted — this function does not distinguish them (same idempotency as del store[k]).

Keys are the caller’s keys: with the store held as an argument the chain is resolved via dol.inner_most_key (alive store — no weakref hole). Validate-the-target caveat (D4): the first argument must be an s3dol store or a dol wrapper over one.

s3dol.diagnose(connection: S3Connection | None = None, *, environ: Mapping[str, str] | None = None, aws_config: Mapping | None = None, file=None, **kwargs) → str

Print (and return) the resolution report. Never raises.

Call it either with a ready S3Connection, with S3Connection kwargs, or — the migration case — with exactly the kwargs you pass s3dol.store.S3Store today (bucket_name=, path=, profile_name=, …), which additionally prints the v0-vs-v1 divergence table.

>>> report = diagnose(endpoint_url='http://localhost:9000',
...                   environ={}, file=io.StringIO())
>>> 'endpoint_url' in report and 'resolution' in report.lower()
True

s3dol.handles(store) → BucketHandles

The sibling store of ObjectHandles: handles(s)[k].

s3dol.info(store) → BucketInfo

The sibling store of ObjectInfo: info(s)[k].

s3dol.prefixes(store) → list

The ‘directories’ one level under the store’s scope, relative to the caller’s key space — one ListObjectsV2(Delimiter='/') reading CommonPrefixes (which v0 parsed and then called from nowhere; ADR-0009). Through a dol wrapper chain, the listing is rooted at the caller’s scope (each prefix layer’s contribution) and results are mapped back outward (_key_of_id per layer, innermost first — the inverse walk dol does not provide; ADR-0011 D3). A wrapper whose codec cannot express the scope raises — loud, not wrong.

s3dol.register_preset(preset: Preset, *, overwrite: bool = False) → None

Add a row. Open-closed: adding a provider is adding a row.

s3dol.resolve(spec: S3Connection, environ: Mapping[str, str], aws_config: Mapping | Callable[[], Mapping] | None = None) → Resolution

Pure resolution: no os.environ, no disk, no network (ADR-0012 D2).

environ is a plain mapping (pass os.environ at real call sites, a dict in tests). aws_config is a pre-loaded, credential-scrubbed config mapping, or a thunk for one (load_aws_config()) — called only if a rung needs it.

s3dol.s3_store(bucket: str, *, prefix: str = '', preset: str | Preset | None = None, connection: S3Connection | None = None, endpoint_url: str | None = None, region_name: str | None = None, profile: str | None = None, credentials=None, anon: bool = False, on_missing_bucket: str = 'assume', readonly: bool = False, value_codec=None, reads: ReadStrategy | None = None, writes: WriteStrategy | None = None)

One bucket as a MutableMapping[str, bytes] (a Mapping when readonly=True).

Annotated (deliberately) as the abstract Mapping, not -> BucketStore: with value_codec the concrete type is a generated dol wrapper class that is not a BucketStore subclass. (v0 annotated its factory -> Store and returned something that was not one; inverting that lie is not an improvement — architecture.md.)

Connection axes (preset/endpoint_url/region_name/profile/ credentials/anon) are conveniences that build an S3Connection; pass connection= instead for full control (they are mutually exclusive with it).

s3dol.sub(store, prefix: str)

A store scoped to prefix, in the caller’s key space (D3).

Two branches, deliberately different (each documented in the ADR):

• unwrapped s3dol store: leaf._with(prefix=...) — cheap, keeps server-side prefix pushdown, returns the same class;

• dol-wrapped store: composes Pipe(filt_iter.prefixes(p), KeyCodecs.prefixed(p)) over the outer store (filter FIRST — the only safe composition, ADR-0006 §1), which preserves the user’s codecs but loses pushdown (a full scan filtered client-side; D8 closed this: the cheap path is the unwrapped branch) and returns a different type.

s3dol.urls(store, *, expires_in: int = 3600, client_method: str = 'get_object') → BucketUrls

The sibling store of presigned URLs: urls(s)[k].