> built 2026-10-14 01:07 UTC from 02cbc8c (master) · xdol 0.1.17. Details: build_info.json
# index.html.md
# xdol
Extended Data Object Layers - dol-based tools
To install: `pip install xdol`
[Documentation](https://i2mint.github.io/xdol/)
## Persistent memoizer (record/replay)
Wrap a costly function (a paid API, a slow render) so the first run records its result to disk and every later run replays it:
```python
from xdol import memoize_calls
@memoize_calls(store="tests/cassettes", mode="record") # default mode is replay-only: a miss raises
def slow_square(x):
return {"result": x * x}
```
Stable keys (`xdol.keys`), text / JSON / binary / paired `.json`+`.bin` codecs, any `MutableMapping` as the store (a folder, S3), spend-safe replay by default. See [misc/docs/persistent_memoizer.md]().
This documentation as a single file: xdol.md (Markdown, for agents).
# _autosummary/xdol.html.md
# xdol
Extended tools for dol (Data Object Layers)
### Modules
| [`keys`](_autosummary/xdol.keys.html.md#module-xdol.keys) | Robust, stable cache keys: turn a call's inputs into a short, filesystem-safe string. |
|--------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------|
| [`memoize`](_autosummary/xdol.memoize.html.md#module-xdol.memoize) | Persistent memoization with record/replay modes, for costly calls. |
| [`paths`](_autosummary/xdol.paths.html.md#module-xdol.paths) | Tools for manipulating paths. |
| [`pystores`](_autosummary/xdol.pystores.html.md#module-xdol.pystores) | Stores for python objects |
| [`registry`](_autosummary/xdol.registry.html.md#module-xdol.registry) | A typed dict-backed plugin registry. |
| [`updating`](_autosummary/xdol.updating.html.md#module-xdol.updating) | mapping using customizable policies to determine which items to update. |
| [`util`](_autosummary/xdol.util.html.md#module-xdol.util) | Utility functions for xdol. |
| [`x_codecs`](_autosummary/xdol.x_codecs.html.md#module-xdol.x_codecs) | Extra codecs: object <-> bytes pairs, and stores that apply them. |
# _autosummary/xdol.keys.html.md
# xdol.keys
Robust, stable cache keys: turn a call’s inputs into a short, filesystem-safe string.
A persistent cache (see [`xdol.memoize`](_autosummary/xdol.memoize.html.md#module-xdol.memoize)) is only as good as its keys. A key must be
- **stable across runs and machines** – never derived from a memory address, set
iteration order or dict insertion order;
- **insensitive to superficial differences** – dict key order, surrounding whitespace;
- **loud about what it cannot make stable** – an object whose only description is
`` would give a *different key every run* (a cache that
never hits, or, in replay-only mode, a spurious miss), so it raises
[`UnstableKeyError`](_autosummary/xdol.keys.html.md#xdol.keys.UnstableKeyError) instead of silently hashing the address.
```pycon
>>> robust_key({"b": 1, "a": 2}, prompt=" hi ") == robust_key({"a": 2, "b": 1}, prompt="hi")
True
>>> len(robust_key(1, 2, x=3))
32
```
Keys are a **stability contract**: committed caches (e.g. cassette directories) are keyed
by these hashes, so the output for JSON-able inputs does not change between releases.
Customize with [`make_key_func()`](_autosummary/xdol.keys.html.md#xdol.keys.make_key_func) – drop non-semantic arguments (callbacks, loggers),
bind to the wrapped function’s signature, pick the digest, or teach it your own types
with a `__cache_key__` method.
```pycon
>>> def f(prompt, *, logger=None, retries=3): ...
>>> key = make_key_func(f, drop=["logger", "retries"])
>>> key("hi", logger=print) == key(prompt="hi", retries=9)
True
```
### Functions
| [`hash_payload`](_autosummary/xdol.keys.html.md#xdol.keys.hash_payload)(payload, \*[, digest, length]) | Hash an already-normalized JSON-able `payload` to a short hex string. |
|---------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
| [`make_key_func`](_autosummary/xdol.keys.html.md#xdol.keys.make_key_func)([func, drop, apply_defaults, ...]) | Build a key function `(*args, **kwargs) -> str`, customized. |
| [`normalize`](_autosummary/xdol.keys.html.md#xdol.keys.normalize)(obj, \*[, collapse_whitespace, ...]) | Recursively reduce `obj` to a JSON-able structure that is equal for equal meaning. |
| [`robust_key`](_autosummary/xdol.keys.html.md#xdol.keys.robust_key)(\*args, \*\*kwargs) | The default key function: a stable SHA-256 (32 hex chars) of the normalized call. |
### Exceptions
| [`UnstableKeyError`](_autosummary/xdol.keys.html.md#xdol.keys.UnstableKeyError) | An input has no run-stable description, so no stable key can be made from it. |
|---------------------------------------------------------------------|---------------------------------------------------------------------------------|
### *exception* xdol.keys.UnstableKeyError
Bases: [`TypeError`](https://docs.python.org/3/builtins/exceptions.html#TypeError)
An input has no run-stable description, so no stable key can be made from it.
Fix it by giving the object a `__cache_key__()` method, dropping the argument from
the key (`make_key_func(f, drop=[...])`), or passing `on_unstable="ignore"` if
you accept a key that may change between runs.
### xdol.keys.hash_payload(payload, , digest='sha256', length=32)
Hash an already-normalized JSON-able `payload` to a short hex string.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
### xdol.keys.make_key_func(func=None, , drop=(), apply_defaults=False, collapse_whitespace=False, on_unstable='raise', digest='sha256', length=32, prefix='')
Build a key function `(*args, **kwargs) -> str`, customized.
* **Parameters:**
* **func** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – if given, calls are bound to its signature first, so `f(1, 2)`,
`f(1, b=2)` and `f(a=1, b=2)` share a key. Required for `drop`.
* **drop** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – names of parameters of `func` to leave out of the key – the
non-semantic ones (callbacks, loggers, timeouts, `self`).
* **apply_defaults** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – with `func`, also hash default values, so `f(x)` equals
`f(x, y=)`. Off by default: adding a defaulted parameter later
then does not invalidate existing caches (changing a default does not
either – turn this on if that matters to you).
* **collapse_whitespace** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – see [`normalize()`](_autosummary/xdol.keys.html.md#xdol.keys.normalize).
* **on_unstable** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'raise'`, `'warn'`, `'ignore'`]) – see [`normalize()`](_autosummary/xdol.keys.html.md#xdol.keys.normalize).
* **digest** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – hash algorithm (any [`hashlib`](https://docs.python.org/3/library/hashlib.html#module-hashlib) name) and key length in hex chars.
* **length** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – hash algorithm (any [`hashlib`](https://docs.python.org/3/library/hashlib.html#module-hashlib) name) and key length in hex chars.
* **prefix** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – prepended to every key (e.g. a namespace or version: `"v2-"`).
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
With no arguments this is [`robust_key()`](_autosummary/xdol.keys.html.md#xdol.keys.robust_key).
```pycon
>>> make_key_func()("x") == robust_key("x")
True
>>> def f(a, b=1, *, log=None): ...
>>> k = make_key_func(f, drop=["log"], apply_defaults=True)
>>> k(1) == k(1, 1) == k(a=1, log=print)
True
```
### xdol.keys.normalize(obj, , collapse_whitespace=False, on_unstable='raise')
Recursively reduce `obj` to a JSON-able structure that is equal for equal meaning.
- strings are stripped (and, with `collapse_whitespace`, inner runs of whitespace
become one space – off by default since whitespace can be meaningful in code or
markdown);
- mappings get string keys, sorted; lists and tuples become lists;
- sets are sorted; bytes are replaced by their digest; paths, enums, dates,
dataclasses, numpy-like arrays and callables get a tagged, stable description;
- an object with a `__cache_key__()` method is described by what it returns;
- pandas objects are described by their full content; bound methods by function and
instance; lambdas raise (indistinguishable);
- anything else falls back to `repr`, vetted by `on_unstable` (see
[`UnstableKeyError`](_autosummary/xdol.keys.html.md#xdol.keys.UnstableKeyError)). A `repr` can still be lossy (truncated, or leaving out
> state): for such types define `__cache_key__`.
```pycon
>>> normalize({"b": " x ", "a": (1, 2)})
{'a': [1, 2], 'b': 'x'}
>>> normalize({3, 1, 2})
{'__set__': [1, 2, 3]}
```
* **Return type:**
[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)
### xdol.keys.robust_key(\*args, \*\*kwargs)
The default key function: a stable SHA-256 (32 hex chars) of the normalized call.
Calls whose arguments are semantically equal – same values up to dict key order and
surrounding whitespace – get the same key.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> robust_key("a", n=1) == robust_key("a ", n=1)
True
>>> robust_key("a", n=1) == robust_key("a", n=2)
False
```
# _autosummary/xdol.memoize.html.md
# xdol.memoize
Persistent memoization with record/replay modes, for costly calls.
A transparent wrapper that persists a function’s results in a `MutableMapping` keyed by
a stable hash of the call. It is a VCR cassette for *any* function, not just HTTP: the
first run records, every later run replays from the store – no call, no cost, no network.
The driving use is tests of code that calls paid APIs (LLMs, image/video generation),
which must stay free and fast, but it serves any expensive, deterministic-enough call.
```pycon
>>> import tempfile
>>> calls = []
>>> @memoize_calls(store=tempfile.mkdtemp(), mode="record")
... def slow_square(x):
... calls.append(x)
... return {"result": x * x}
>>> slow_square(3), slow_square(3)
({'result': 9}, {'result': 9})
>>> calls
[3]
```
**Replay is the default; recording is an opt-in.** Under the default `mode=None` a call
whose key is not in the store raises [`CacheMiss`](_autosummary/xdol.memoize.html.md#xdol.memoize.CacheMiss) instead of calling the function,
unless the run sets the `XDOL_MEMOIZE_RECORD` environment variable (see
[`recording_enabled()`](_autosummary/xdol.memoize.html.md#xdol.memoize.recording_enabled)). A key that quietly moved would otherwise read as a normal,
billable test run. Spending is an act that carries a flag. For a plain memoizer that
records on a miss, pass `mode="record"` (as above).
Four pieces, each a keyword argument with a good default (a *seam*):
- `key_func` – input to key; default [`xdol.keys.robust_key()`](_autosummary/xdol.keys.html.md#xdol.keys.robust_key) (see also `drop=`);
- `codec` – result to bytes; `"json"` (default), `"text"`, `"bytes"` for binary
assets, or `"paired"` for a `(metadata, asset_bytes)` result stored as
`.json` + `.bin` (see [`xdol.x_codecs`](_autosummary/xdol.x_codecs.html.md#module-xdol.x_codecs));
- `store` – a directory path (a stable, committable folder: see
[`file_cache_store()`](_autosummary/xdol.memoize.html.md#xdol.memoize.file_cache_store)), or any `MutableMapping`. Pass a bytes store such as
> `s3dol.s3_store(bucket)` and the codec is applied on top, so swapping the filesystem
> for S3 changes nothing at the call site;
- `mode` – `"replay"`, `"record"`, `"refresh"` or `"off"`; see `MODES`.
**Cache the top-level composite**: wrap the whole `prompt -> JSON -> asset URL ->
downloaded bytes` pipeline, not each intermediate API step, so one intent maps to one
final result. Nothing enforces this; it keeps the surface area (and the number of
cassettes to review) small.
The wrapper preserves the function’s signature and metadata (`functools.wraps`) and
supports coroutine functions. Exceptions raised by the function are never cached.
### Module Attributes
| [`RECORD_ENV`](_autosummary/xdol.memoize.html.md#xdol.memoize.RECORD_ENV) | Set to a truthy value to let `mode=None` calls record on a miss. |
|----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`TRUTHY_ENV_VALUES`](_autosummary/xdol.memoize.html.md#xdol.memoize.TRUTHY_ENV_VALUES) | Environment-variable spellings that count as "on". |
| [`DFLT_CORRUPT_ERRORS`](_autosummary/xdol.memoize.html.md#xdol.memoize.DFLT_CORRUPT_ERRORS) | Errors that mean "this stored entry cannot be decoded" (as opposed to a store/network failure, which must never be mistaken for a missing entry and cost a real call). |
### Functions
| [`file_cache_store`](_autosummary/xdol.memoize.html.md#xdol.memoize.file_cache_store)(rootdir, \*[, codec]) | A persistent, atomic, codec-aware `MutableMapping` over a directory. |
|----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------|
| [`memoize_calls`](_autosummary/xdol.memoize.html.md#xdol.memoize.memoize_calls)([func, codec, key_func, drop, ...]) | Persistently memoize `func` into `store` (record/replay). |
| [`recording_enabled`](_autosummary/xdol.memoize.html.md#xdol.memoize.recording_enabled)([env]) | True when the environment variable `env` is set to a truthy value. |
### Classes
| [`AtomicFiles`](_autosummary/xdol.memoize.html.md#xdol.memoize.AtomicFiles)(\*args[, delete_func]) | A bytes store over a directory whose writes are atomic. |
|---------------------------------------------------------------------------------------|-----------------------------------------------------------|
### Exceptions
| [`CacheCorrupt`](_autosummary/xdol.memoize.html.md#xdol.memoize.CacheCorrupt) | A stored entry exists but cannot be decoded (replay mode refuses to guess). |
|-----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| [`CacheMiss`](_autosummary/xdol.memoize.html.md#xdol.memoize.CacheMiss) | A memoized call missed its store while recording was disabled. |
| [`CacheWriteError`](_autosummary/xdol.memoize.html.md#xdol.memoize.CacheWriteError)(message, result) | The function ran but its result could not be stored (e.g. not JSON-serializable). |
### *class* xdol.memoize.AtomicFiles(\*args, delete_func=, \*\*kwargs)
Bases: `Files`
A bytes store over a directory whose writes are atomic.
Plain file writes can leave a truncated cassette if the process dies mid-write, which
then fails to decode (or worse, decodes). Here each value is written to a temporary
sibling and renamed into place (`os.replace`), and temporary files are invisible to
iteration. Missing sub-directories of a key (`"a/b"`) are created.
### *exception* xdol.memoize.CacheCorrupt
Bases: [`RuntimeError`](https://docs.python.org/3/builtins/exceptions.html#RuntimeError)
A stored entry exists but cannot be decoded (replay mode refuses to guess).
### *exception* xdol.memoize.CacheMiss
Bases: [`LookupError`](https://docs.python.org/3/builtins/exceptions.html#LookupError)
A memoized call missed its store while recording was disabled.
The alternative to this exception is a real, possibly billable call nobody asked
for. Two things cause it: an input the store has genuinely never seen, and a cache
**key** that moved under an unchanged input – usually because the key was computed
over something incidental (a rendered global registry, a timestamp, a path).
### *exception* xdol.memoize.CacheWriteError(message, result)
Bases: [`RuntimeError`](https://docs.python.org/3/builtins/exceptions.html#RuntimeError)
The function ran but its result could not be stored (e.g. not JSON-serializable).
The result is not lost: it is the `result` attribute.
### xdol.memoize.DFLT_CORRUPT_ERRORS *= (,)*
Errors that mean “this stored entry cannot be decoded” (as opposed to a store/network
failure, which must never be mistaken for a missing entry and cost a real call).
### xdol.memoize.RECORD_ENV *= 'XDOL_MEMOIZE_RECORD'*
Set to a truthy value to let `mode=None` calls record on a miss.
### xdol.memoize.TRUTHY_ENV_VALUES *= frozenset({'1', 'on', 'true', 'yes'})*
Environment-variable spellings that count as “on”.
### xdol.memoize.file_cache_store(rootdir, , codec='json')
A persistent, atomic, codec-aware `MutableMapping` over a directory.
Keys are filenames, values are decoded objects. `rootdir` is
created if missing and is meant to be a stable folder you commit (a cassette
directory), not a temp dir. `codec="paired"` stores `(metadata, asset_bytes)`
values as `.json` + `.bin`.
* **Return type:**
[`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)
```pycon
>>> import tempfile
>>> store = file_cache_store(tempfile.mkdtemp())
>>> store["k"] = {"a": 1}
>>> store["k"], list(store)
({'a': 1}, ['k'])
```
### xdol.memoize.memoize_calls(func=None, \*, store, codec=None, key_func=None, drop=(), mode=None, default_mode='replay', record_env='XDOL_MEMOIZE_RECORD', corrupt_errors=(, ), on_write_error='raise', copy_results=None, reread_after_write=True)
Persistently memoize `func` into `store` (record/replay).
Usable as `memoize_calls(fn, store=...)` or as a decorator
`@memoize_calls(store=...)`.
* **Parameters:**
* **func** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the (costly) function; wrap the *composite* top-level call.
* **store** – a directory path (-> [`file_cache_store()`](_autosummary/xdol.memoize.html.md#xdol.memoize.file_cache_store)), or a `MutableMapping`.
* **codec** – `None` (default) means a path gets `"json"` and a mapping is used as
it is (its values are the results). Otherwise `store` is a *bytes* mapping
and this is applied on top: `"json"`, `"text"`, `"bytes"`,
`"paired"`, a [`xdol.x_codecs.Codec`](_autosummary/xdol.x_codecs.html.md#xdol.x_codecs.Codec), or an `(encode, decode)` pair.
* **key_func** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – input to key. Default [`xdol.keys.robust_key()`](_autosummary/xdol.keys.html.md#xdol.keys.robust_key).
* **drop** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – parameter names of `func` to leave out of the key (callbacks, loggers,
`self`); builds the key with [`xdol.keys.make_key_func()`](_autosummary/xdol.keys.html.md#xdol.keys.make_key_func), which also
binds calls to the signature. Not combinable with `key_func`.
* **mode** (`Union`[[`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'replay'`, `'record'`, `'refresh'`, `'off'`], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[], [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'replay'`, `'record'`, `'refresh'`, `'off'`]], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – one of `MODES`, a zero-argument callable returning one (evaluated on
every call), or `None` (default) for: `"record"` if the `record_env`
variable is truthy, else `default_mode`. Read at call time, so setting the
variable after import (e.g. in a pytest hook) works.
* **default_mode** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'replay'`, `'record'`, `'refresh'`, `'off'`]) – what `mode=None` means when the environment does not opt in.
`"replay"` (default) makes spending an explicit act.
* **record_env** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – name of the opt-in environment variable.
* **corrupt_errors** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`type`](https://docs.python.org/3/builtins/functions.html#type)[[`BaseException`](https://docs.python.org/3/builtins/exceptions.html#BaseException)], [`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis)]) – exception types raised by reading an entry that mean “undecodable”
(default: [`xdol.x_codecs.CodecDecodeError`](_autosummary/xdol.x_codecs.html.md#xdol.x_codecs.CodecDecodeError), which the codecs raise;
add your own for an object-valued store that can fail to load). In replay
mode they raise [`CacheCorrupt`](_autosummary/xdol.memoize.html.md#xdol.memoize.CacheCorrupt); when recording, the entry is recomputed
and overwritten, with a warning. Any other error – a network failure, a
misconfigured store – propagates and never costs a real call.
* **copy_results** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – deep-copy values read from the store, so callers cannot mutate the
cache. `None` (default) means: yes when `store` is an object-valued
mapping used as is (a codec makes a fresh object per read anyway).
* **reread_after_write** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – after recording, return the value as read back from the store,
so a recording run and a replay run return *the same thing* (JSON turns tuples
into lists and int keys into strings). Turn off to skip the extra read of a
large asset.
* **on_write_error** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'raise'`, `'warn'`]) – what to do if a fresh result cannot be stored: `"raise"`
[`CacheWriteError`](_autosummary/xdol.memoize.html.md#xdol.memoize.CacheWriteError) (carrying the result) or `"warn"` and return it.
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)
The returned wrapper has attributes `cache_store`, `key_func`, `cache_key(\*args,
\*\*kwargs)` and `cache_mode()` (the mode that would apply now).
### xdol.memoize.recording_enabled(env='XDOL_MEMOIZE_RECORD')
True when the environment variable `env` is set to a truthy value.
The opt-in is environment-level rather than per-call, so “this run may spend” is one
visible fact about the run instead of a property of each test.
* **Return type:**
[`bool`](https://docs.python.org/3/builtins/functions.html#bool)
```pycon
>>> recording_enabled("XDOL_SURELY_UNSET_VARIABLE")
False
```
# _autosummary/xdol.paths.html.md
# xdol.paths
Tools for manipulating paths.
A path is a sequence of objects that represent a reference or location.
Often paths are strings that use a special character (e.g. ‘/’) to distinguish the
separate objects in the sequence.
### Functions
| [`extract_path_segments`](_autosummary/xdol.paths.html.md#xdol.paths.extract_path_segments)(path, \*[, sep, depth, ...]) | Extracts the first `depth` segments of a path, delimited by `sep`. |
|-------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|
### xdol.paths.extract_path_segments(path, , sep='/', depth=1, trailing_sep=True)
Extracts the first `depth` segments of a path, delimited by `sep`.
* **Parameters:**
* **path** – The path to extract the segments from.
* **sep** – The delimiter to use to separate the segments.
* **depth** – The number of segments to extract.
* **trailing_sep** – Whether to add a trailing delimiter to the result.
* **Returns:**
The first `depth` segments of the path, delimited by `sep`.
By default, the function will use depth=1 and the system’s path separator as `sep`.
```pycon
>>> extract_path_segments("snap/crackle/pop/cereal", sep='/')
'snap/'
```
You can specify a different `depth`.
```pycon
>>> extract_path_segments("snap/crackle/pop/cereal", depth=2, sep='/')
'snap/crackle/'
```
Note that if the path starts with the `sep` delimiter, the result will be different.
```pycon
>>> extract_path_segments("/snap/crackle/pop/cereal", depth=2, sep='/')
'/snap/'
```
If `path` is not a string, the function will assume it’s an iterable of strings,
and return an iterable of the results.
```pycon
>>> list(extract_path_segments(
... ['a.file', 'a/folder', 'a/folder_with/subfolders/', 'another/folder'],
... depth=2, sep='/'
... ))
['a.file', 'a/folder', 'a/folder_with/', 'another/folder']
```
You can of course specify a different delimiter.
```pycon
>>> extract_path_segments("snap.crackle.pop.cereal", sep='.', depth=2)
'snap.crackle.'
```
That last dot doesn’t make sense here? Tell the function to not add the separator
at the end of the result.
```pycon
>>> extract_path_segments(
... "snap.crackle.pop.cereal", sep='.', depth=2, trailing_sep=False
... )
'snap.crackle'
```
# _autosummary/xdol.pystores.html.md
# xdol.pystores
Stores for python objects
### Functions
| `bytes_decoder`(x) | |
|----------------------|----|
### Classes
| [`PkgFilesReader`](_autosummary/xdol.pystores.html.md#xdol.pystores.PkgFilesReader) | |
|-----------------------------------------------------------------------------------------|----------------------------------------------------------------|
| [`PkgReader`](_autosummary/xdol.pystores.html.md#xdol.pystores.PkgReader)(rootdir[, subpath, ...]) | By store we mean key-value store. |
| [`PyFilesBytes`](_autosummary/xdol.pystores.html.md#xdol.pystores.PyFilesBytes)(rootdir[, subpath, ...]) | Mapping interface to .py files' bytes |
| [`PyFilesReader`](_autosummary/xdol.pystores.html.md#xdol.pystores.PyFilesReader)([store]) | Mapping interface to .py files of a folder. |
| [`PyprojectReader`](_autosummary/xdol.pystores.html.md#xdol.pystores.PyprojectReader)(src, \*[, max_levels]) | Mapping interface to pyproject.toml files in a directory tree. |
| [`SetupCfgReader`](_autosummary/xdol.pystores.html.md#xdol.pystores.SetupCfgReader)(src, \*[, max_levels]) | Mapping interface to setup.cfg files in a directory tree. |
### xdol.pystores.PkgFilesReader
alias of [`PyFilesReader`](_autosummary/xdol.pystores.html.md#xdol.pystores.PyFilesReader)
### *class* xdol.pystores.PkgReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False, allow_escape=False)
Bases: `Store`
By store we mean key-value store. This could be files in a filesystem, objects in s3, or a database. Where and
how the content is stored should be specified, but StoreInterface offers a dict-like interface to this.
```default
__getitem__ calls: _id_of_key _obj_of_data
__setitem__ calls: _id_of_key _data_of_obj
__delitem__ calls: _id_of_key
__iter__ calls: _key_of_id
```
```pycon
>>> # Default store: no key or value conversion #####################################
>>> from dol import Store
>>> s = Store()
>>> s['foo'] = 33
>>> s['bar'] = 65
>>> assert list(s.items()) == [('foo', 33), ('bar', 65)]
>>> assert list(s.store.items()) == [('foo', 33), ('bar', 65)] # see that the store contains the same thing
>>>
>>> #################################################################################
>>> # Now let's make stores that have a key and value conversion layer ##############
>>> # input keys will be upper cased, and output keys lower cased ###################
>>> # input values (assumed int) will be converted to ascii string, and visa versa ##
>>> #################################################################################
>>>
>>> def test_store(s):
... s['foo'] = 33 # write 33 to 'foo'
... assert 'foo' in s # __contains__ works
... assert 'no_such_key' not in s # __nin__ works
... s['bar'] = 65 # write 65 to 'bar'
... assert len(s) == 2 # there are indeed two elements
... assert list(s) == ['foo', 'bar'] # these are the keys
... assert list(s.keys()) == ['foo', 'bar'] # the keys() method works!
... assert list(s.values()) == [33, 65] # the values() method works!
... assert list(s.items()) == [('foo', 33), ('bar', 65)] # these are the items
... assert list(s.store.items()) == [('FOO', '!'), ('BAR', 'A')] # but note the internal representation
... assert s.get('foo') == 33 # the get method works
... assert s.get('no_such_key', 'something') == 'something' # return a default value
... del(s['foo']) # you can delete an item given its key
... assert len(s) == 1 # see, only one item left!
... assert list(s.items()) == [('bar', 65)] # here it is
>>>
>>> # We can introduce this conversion layer in several ways. Here's a few... ######################
>>> # by subclassing ###############################################################################
>>> class MyStore(Store):
... def _id_of_key(self, k):
... return k.upper()
... def _key_of_id(self, _id):
... return _id.lower()
... def _data_of_obj(self, obj):
... return chr(obj)
... def _obj_of_data(self, data):
... return ord(data)
>>> s = MyStore(store=dict()) # note that you don't need to specify dict(), since it's the default
>>> test_store(s)
>>>
>>> # by assigning functions to converters ##########################################################
>>> class MyStore(Store):
... def __init__(self, store, _id_of_key, _key_of_id, _data_of_obj, _obj_of_data):
... super().__init__(store)
... self._id_of_key = _id_of_key
... self._key_of_id = _key_of_id
... self._data_of_obj = _data_of_obj
... self._obj_of_data = _obj_of_data
...
>>> s = MyStore(dict(),
... _id_of_key=lambda k: k.upper(),
... _key_of_id=lambda _id: _id.lower(),
... _data_of_obj=lambda obj: chr(obj),
... _obj_of_data=lambda data: ord(data))
>>> test_store(s)
>>>
>>> # using a Mixin class #############################################################################
>>> class Mixin:
... def _id_of_key(self, k):
... return k.upper()
... def _key_of_id(self, _id):
... return _id.lower()
... def _data_of_obj(self, obj):
... return chr(obj)
... def _obj_of_data(self, data):
... return ord(data)
...
>>> class MyStore(Mixin, Store): # note that the Mixin must come before Store in the mro
... pass
...
>>> s = MyStore() # no dict()? No, because default anyway
>>> test_store(s)
>>>
>>> # adding wrapper methods to an already made Store instance #########################################
>>> s = Store(dict())
>>> s._id_of_key=lambda k: k.upper()
>>> s._key_of_id=lambda _id: _id.lower()
>>> s._data_of_obj=lambda obj: chr(obj)
>>> s._obj_of_data=lambda data: ord(data)
>>> test_store(s)
```
Note on defining your own “Mapping Views”.
When you do a `.keys()`, a `.values()` or `.items()` you’re getting a `MappingView`
instance; an iterable and sized container that provides some methods to access
particular aspects of the wrapped mapping.
If you need to customize the behavior of these instances, you should avoid
overriding the `keys`, `values` or `items` methods directly, but instead
override the `KeysView`, `ValuesView` or `ItemsView` classes that they use.
For more, see: [https://github.com/i2mint/dol/wiki/Mapping-Views](https://github.com/i2mint/dol/wiki/Mapping-Views)
#### get(k) → D[k] if k in D, else d. d defaults to None.
#### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs)
`is_valid_key` on the inner key – see `mk_relative_path_store`.
#### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs)
`validate_key` on the inner key – see `mk_relative_path_store`.
### *class* xdol.pystores.PyFilesBytes(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False, allow_escape=False)
Bases: `Store`
Mapping interface to .py files’ bytes
#### get(k) → D[k] if k in D, else d. d defaults to None.
#### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs)
`is_valid_key` on the inner key – see `mk_relative_path_store`.
#### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs)
`validate_key` on the inner key – see `mk_relative_path_store`.
### *class* xdol.pystores.PyFilesReader(store=)
Bases: `PrefixRelativizationMixin`, `Store`
Mapping interface to .py files of a folder.
Keys are relative .py paths.
Values are the string contents of the .py file.
Important Note: If the byte contents of the .py file can’t be decoded (with a simple bytes.decode()),
an empty string will be returned as it’s value (i.e. contents).
```pycon
>>> import asyncio
>>> s = PyFilesReader(asyncio)
>>> assert len(s) > 10 # to test length (currently asyncio has 29 modules
>>> 'locks.py' in s
True
```
But you can also specify an \_\_init_\_.py filepath or the directory containing it.
```pycon
>>> import os
>>> init_filepath = asyncio.__file__
>>> dirpath_to_asyncio_modules = os.path.dirname(init_filepath)
>>> ss = PyFilesReader(init_filepath)
>>> sss = PyFilesReader(dirpath_to_asyncio_modules)
>>> assert list(s) == list(ss) == list(sss)
```
#### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs)
`is_valid_key` on the inner key – see `mk_relative_path_store`.
#### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs)
`validate_key` on the inner key – see `mk_relative_path_store`.
### *class* xdol.pystores.PyprojectReader(src, , max_levels=None)
Bases: `Store`
Mapping interface to pyproject.toml files in a directory tree.
Keys are relative paths to pyproject.toml files.
Values are the string contents of the pyproject.toml files.
```pycon
>>> import tempfile, os
>>> with tempfile.TemporaryDirectory() as tmpdir:
... # Create a pyproject.toml file
... proj1_dir = os.path.join(tmpdir, 'proj1')
... os.makedirs(proj1_dir)
... toml_path = os.path.join(proj1_dir, 'pyproject.toml')
... with open(toml_path, 'w') as f:
... _ = f.write('[project]\ndependencies = ["requests>=2.0"]')
...
... reader = PyprojectReader(tmpdir)
... assert len(reader) == 1
... assert 'proj1/pyproject.toml' in reader
... assert 'requests' in reader['proj1/pyproject.toml']
True
```
#### get(k) → D[k] if k in D, else d. d defaults to None.
#### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs)
`is_valid_key` on the inner key – see `mk_relative_path_store`.
#### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs)
`validate_key` on the inner key – see `mk_relative_path_store`.
### *class* xdol.pystores.SetupCfgReader(src, , max_levels=None)
Bases: `Store`
Mapping interface to setup.cfg files in a directory tree.
Keys are relative paths to setup.cfg files.
Values are the string contents of the setup.cfg files.
```pycon
>>> import tempfile, os
>>> with tempfile.TemporaryDirectory() as tmpdir:
... # Create some setup.cfg files
... proj1_dir = os.path.join(tmpdir, 'proj1')
... os.makedirs(proj1_dir)
... cfg_path = os.path.join(proj1_dir, 'setup.cfg')
... with open(cfg_path, 'w') as f:
... _ = f.write('[options]\ninstall_requires = requests')
...
... reader = SetupCfgReader(tmpdir)
... assert len(reader) == 1
... assert 'proj1/setup.cfg' in reader
... assert 'requests' in reader['proj1/setup.cfg']
True
```
#### get(k) → D[k] if k in D, else d. d defaults to None.
#### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs)
`is_valid_key` on the inner key – see `mk_relative_path_store`.
#### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs)
`validate_key` on the inner key – see `mk_relative_path_store`.
# _autosummary/xdol.registry.html.md
# xdol.registry
A typed dict-backed plugin registry.
The [`Registry`](_autosummary/xdol.registry.html.md#xdol.registry.Registry) is the dict-pattern that recurs across packages
(`falaw.registry`, `lacing.processors`, `lookbook.registry`, …): a
dict, a `register(key, value)` call, a `get(key)` lookup, an iteration
protocol, and a small number of conveniences that each package keeps
re-implementing — aliases, lazy loading, decorator-form registration, and
conflict policies. `Registry` collects these into one builtin-only,
`MutableMapping`-compatible class.
Basic use:
```pycon
>>> from xdol.registry import Registry
>>> tools = Registry[str](name="tools")
>>> tools.register("hello", "say hi")
'say hi'
>>> tools["hello"]
'say hi'
>>> sorted(tools)
['hello']
```
Decorator form:
```pycon
>>> handlers = Registry()
>>> @handlers.register_decorator("greet")
... def greet(name): return f"hi {name}"
>>> handlers["greet"]("ada")
'hi ada'
```
Aliases:
```pycon
>>> models = Registry()
>>> models.register("fal-ai/flux/dev", {"family": "flux"})
{'family': 'flux'}
>>> models.alias("flux", "fal-ai/flux/dev")
>>> models["flux"]
{'family': 'flux'}
>>> "flux" in models
True
```
Conflict policies — `error` (default), `replace`, `keep`:
```pycon
>>> r = Registry(on_conflict="error")
>>> r.register("k", 1)
1
>>> r.register("k", 2)
Traceback (most recent call last):
...
xdol.registry.RegistryConflict: 'k' is already registered in
```
```pycon
>>> r2 = Registry(on_conflict="keep")
>>> r2.register("k", 1)
1
>>> r2.register("k", 2) # silently kept
1
>>> r2["k"]
1
```
Plain assignment replaces; opt in to `strict_setitem` to make it honour the policy:
```pycon
>>> loose = Registry()
>>> loose["k"] = 1
>>> loose["k"] = 2 # explicit assignment replaces
>>> loose["k"]
2
>>> strict = Registry(strict_setitem=True)
>>> strict["k"] = 1
>>> strict["k"] = 2
Traceback (most recent call last):
...
xdol.registry.RegistryConflict: 'k' is already registered in
```
Lazy loading — value is built on first access:
```pycon
>>> calls = []
>>> def make_thing():
... calls.append(1)
... return {"built": True}
>>> r3 = Registry()
>>> r3.register_lazy("thing", make_thing)
>>> calls
[]
>>> r3["thing"]
{'built': True}
>>> r3["thing"] # cached after first build
{'built': True}
>>> calls
[1]
```
Subscription — observe registrations as they happen:
```pycon
>>> r4 = Registry()
>>> seen = []
>>> sub = r4.subscribe(lambda key, value: seen.append((key, value)))
>>> r4.register("a", 1)
1
>>> r4.register("b", 2)
2
>>> seen
[('a', 1), ('b', 2)]
>>> sub.unsubscribe()
>>> r4.register("c", 3)
3
>>> seen # no longer notified
[('a', 1), ('b', 2)]
```
### Classes
| [`Registry`](_autosummary/xdol.registry.html.md#xdol.registry.Registry)(\*[, name, on_conflict, strict_setitem]) | A typed dict-backed plugin registry. |
|------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
| [`Subscription`](_autosummary/xdol.registry.html.md#xdol.registry.Subscription)(registry, callback) | Handle returned by [`Registry.subscribe()`](_autosummary/xdol.registry.html.md#xdol.registry.Registry.subscribe). |
### Exceptions
| [`RegistryConflict`](_autosummary/xdol.registry.html.md#xdol.registry.RegistryConflict) | Raised when registering a key that is already taken under `on_conflict='error'`. |
|---------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|
| [`RegistryError`](_autosummary/xdol.registry.html.md#xdol.registry.RegistryError) | Base for registry errors. |
| [`RegistryMissing`](_autosummary/xdol.registry.html.md#xdol.registry.RegistryMissing) | Raised by [`Registry.get()`](_autosummary/xdol.registry.html.md#xdol.registry.Registry.get) when a key is missing. |
### *class* xdol.registry.Registry(, name='', on_conflict='error', strict_setitem=False)
Bases: [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping), [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic)[`V`]
A typed dict-backed plugin registry.
* **Parameters:**
* **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Optional name; appears in error messages and `repr`.
* **on_conflict** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'error'`, `'replace'`, `'keep'`]) – `'error'` (default) raises [`RegistryConflict`](_autosummary/xdol.registry.html.md#xdol.registry.RegistryConflict) when registering
a key that already exists. `'replace'` silently overwrites.
`'keep'` silently keeps the original.
* **strict_setitem** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Opt-in (default `False`). By default `reg[k] = v` is an explicit
assignment that replaces whatever is there, bypassing `on_conflict`.
With `strict_setitem=True` it follows the `on_conflict` policy exactly
like [`register()`](_autosummary/xdol.registry.html.md#xdol.registry.Registry.register) (so `MutableMapping.update` and `setdefault`
do too; note `update` is not atomic: entries before the first conflict stay).
* **(****reading** (*Keys that are aliases resolve to their canonical key everywhere*)
:param :
:param `reg[k] = v`:
:param [`register()`](_autosummary/xdol.registry.html.md#xdol.registry.Registry.register):
:param [`register_lazy()`](_autosummary/xdol.registry.html.md#xdol.registry.Registry.register_lazy)):
:param so the `on_conflict`:
:param policy always applies to the canonical entry.:
:param Implements `MutableMapping` so anything that takes a mapping (`dict(...)`:
:param :
:param iteration:
:param length checks:
:param `in` lookups:
:param `.items()`) just works.:
#### alias(alias, target)
Make `alias` resolve to `target` (which must already be registered).
* **Return type:**
[`None`](https://docs.python.org/3/builtins/constants.html#None)
#### get(key, default=