xdol.x_codecs#

Extra codecs: object <-> bytes pairs, and stores that apply them.

Besides resolve_values_to_bytes(), this module holds the value-encoding half of the persistent memoizer (xdol.memoize):

Codec – an encode/decode pair, with ready-made JSON_CODEC (human-readable, diff-friendly), TEXT_CODEC and BYTES_CODEC (pass-through, for images/audio), looked up by name through get_codec();
with_codec() – apply a codec to the values of a bytes store;
paired_store() – for a result that mixes metadata and a binary asset, a store whose value is (metadata, asset_bytes), persisted as <key>.json + <key>.bin.

>>> store = with_codec({}, "json")
>>> store["k"] = {"b": 1, "a": [1, 2]}
>>> store["k"]
{'a': [1, 2], 'b': 1}
>>> raw = {}
>>> pairs = paired_store(raw)
>>> pairs["img1"] = ({"prompt": "a cat"}, b"PNGDATA")
>>> sorted(raw)
['img1.bin', 'img1.json']
>>> pairs["img1"]
({'prompt': 'a cat'}, b'PNGDATA')

Module Attributes

`JSON_CODEC`	Text / JSON results, stored human-readably so a committed cache is reviewable.
`TEXT_CODEC`	`str` results, stored as UTF-8.
`BYTES_CODEC`	Pass-through, for binary assets (image / audio bytes).
`CODECS`	`CODECS["pickle"] = Codec(pickle.dumps, pickle.loads)`.

Functions

`get_codec`(codec)	Resolve a `Codec`, a name in `CODECS`, or an `(encode, decode)` pair.
`json_codec`(*[, indent, default])	A JSON `Codec`: keys sorted, non-ASCII kept, so stored files diff well.
`paired_store`(store, **kwargs)	See `PairedStore`.
`with_codec`(store[, codec])	Wrap a bytes `store` so its values are objects, encoded/decoded by `codec`.

Classes

`Codec`(encode, decode)	An object <-> bytes pair, for the values of a store.
`PairedStore`(store, *[, meta_codec, ...])	Values are `(metadata, asset_bytes)` pairs, kept as two entries in a bytes store.

Exceptions

CodecDecodeError

A stored value could not be decoded (corrupt or truncated entry, wrong codec).

xdol.x_codecs.BYTES_CODEC = (<function _as_bytes>, <function <lambda>>)#: Pass-through, for binary assets (image / audio bytes).

xdol.x_codecs.CODECS: dict[str, Codec] = {'bytes': (<function _as_bytes>, <function <lambda>>), 'json': (<function json_codec.<locals>.encode>, <function json_codec.<locals>.<lambda>>), 'text': (<function <lambda>>, <function <lambda>>)}#

CODECS["pickle"] = Codec(pickle.dumps, pickle.loads).

Type:: Named codecs. Register your own

class xdol.x_codecs.Codec(encode, decode)[source]#

Bases: NamedTuple

An object <-> bytes pair, for the values of a store.

decode: Callable[[bytes], Any]#: Alias for field number 1

encode: Callable[[Any], bytes]#: Alias for field number 0

exception xdol.x_codecs.CodecDecodeError[source]#

Bases: ValueError

A stored value could not be decoded (corrupt or truncated entry, wrong codec).

Raised by with_codec() and PairedStore in place of whatever the decoder raised (chained as __cause__), so callers can tell “this entry is undecodable” apart from store, network or configuration failures.

xdol.x_codecs.JSON_CODEC = (<function json_codec.<locals>.encode>, <function json_codec.<locals>.<lambda>>)#: Text / JSON results, stored human-readably so a committed cache is reviewable.

class xdol.x_codecs.PairedStore(store, *, meta_codec='json', meta_suffix='.json', asset_suffix='.bin')[source]#

Bases: MutableMapping

Values are (metadata, asset_bytes) pairs, kept as two entries in a bytes store.

store[k] = (meta, blob) writes k + asset_suffix (blob verbatim) and k + meta_suffix ({"asset_sha256": ..., "meta": meta} through meta_codec). The metadata entry is written last and is what makes a key exist, so a crash mid-write leaves no half-entry that reads back as valid; in/iteration/len only see keys whose both halves are present. The digest guards against two writers of the same key interleaving (A’s metadata with B’s asset): such a pair raises CodecDecodeError on read instead of returning the chimera. Entries in the older bare-metadata layout are still read (unverified).

xdol.x_codecs.TEXT_CODEC = (<function <lambda>>, <function <lambda>>)#: str results, stored as UTF-8.

xdol.x_codecs.get_codec(codec)[source]#

Resolve a Codec, a name in CODECS, or an (encode, decode) pair.

Return type:: Codec

>>> get_codec("json") is JSON_CODEC
True
>>> get_codec((str.encode, bytes.decode)).decode(b"x")
'x'

xdol.x_codecs.json_codec(*, indent=2, default=None)[source]#

A JSON Codec: keys sorted, non-ASCII kept, so stored files diff well.

Strict by default: a value JSON cannot represent raises TypeError rather than being silently stringified. Pass default= (as in json.dumps()) to opt in.

Return type:: Codec

>>> c = json_codec(indent=None)
>>> c.decode(c.encode({"b": "é", "a": 1}))
{'a': 1, 'b': 'é'}
>>> c.encode({"b": 1, "a": 2})
b'{"a": 2, "b": 1}'

xdol.x_codecs.paired_store(store, **kwargs)[source]#

See PairedStore. store is a bytes MutableMapping (files, S3, …).

Return type:: PairedStore

xdol.x_codecs.with_codec(store, codec='json')[source]#

Wrap a bytes store so its values are objects, encoded/decoded by codec.

A value that fails to decode raises CodecDecodeError.

Return type:: MutableMapping