dol.atomic

Atomic file writes: write to a sibling temp file, then os.replace it into place.

A crash (or an exception while producing the data) leaves the old content or the new, never a torn or empty file. Dependency-free; atomic_write is the entry point, and atomic_delete its counterpart for removal, write_content_addressed its variant for a name computed from the content. A write killed mid-way leaves its temp file behind; interrupted_writes lists them (TEMP_NAME_PATTERN is their name).

On Windows, a reader holding the destination open makes os.replace and os.unlink fail with PermissionError until it closes (Python’s open does not share delete). Both functions therefore retry that briefly, through a Retry policy (its retry argument): by default Windows’ PermissionError only, ~0.6 s in all. Elsewhere the same error is a real permission problem, so nothing is retried.

file_lock is its companion: an atomic replace is only safe against concurrent appenders and readers when they all hold the same advisory lock. LockedAppender appends under that lock and follows the file when an atomic replace swaps it.

>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), "f.txt")
>>> atomic_write(path, b"hello")
>>> open(path).read(), os.listdir(os.path.dirname(path))
('hello', ['f.txt'])
>>> atomic_write(path, (c.encode() for c in "ab"))  # chunks are streamed
>>> open(path).read()
'ab'

Module Attributes

SHARING_VIOLATIONS

Windows' sharing violation (another handle has the file open); nothing elsewhere, where PermissionError means a real permission problem.

TEMP_NAME_PATTERN

Matches (fullmatch) every name temp_name() makes, and so the temp file a killed atomic_write() leaves behind.

DFLT_KEEP_HANDLE

Whether a LockedAppender keeps its file open between appends.

Functions

atomic_delete(path, *[, retry])

Remove the file path: retry(os.unlink, path), the delete counterpart of atomic_write().

atomic_write(path, data, *[, fsync, perms, ...])

Write data to path atomically (temp file in the same folder, then os.replace).

file_lock(path, *[, sidecar, poll_interval])

Hold an exclusive advisory lock on path's sidecar (see FileLock); yields whether it is held.

interrupted_writes(folder, *[, pattern, ...])

The temp files writes killed mid-way left in folder: sorted absolute paths (str; .. is not collapsed, so each names the file found) of the regular files (never a symlink or folder) whose name fully matches pattern (default: atomic_write()'s temps; pass another writer's, e.g. r"\.blob-.+\.part").

temp_name(name)

The name atomic_write() stages a write to name under: a hidden sibling .<name, clipped to 100 bytes>.<8 random hex>.tmp, always matched by TEMP_NAME_PATTERN.

update_json(path, change, *[, default, ...])

Apply change to the JSON document at path under its lock; return what change returned.

write_content_addressed(folder, data, *[, ...])

Write data into folder under a name computed from its content; return that name (the file is os.path.join(folder, name)).

Classes

FileLock(path, *[, sidecar, poll_interval])

A reusable, exclusive, advisory lock on a sidecar file <path><sidecar> (a shortened hidden name instead, for names near the 255-byte limit).

LockedAppender(path, *[, lock, line_end, ...])

Appends to path under a FileLock, following the file when an atomic_write() (made under the same lock, e.g. a compaction) replaces it.

Retry([on, attempts, backoff_s])

Call op(*args), retrying on on exceptions with doubling backoff.

Exceptions

InvalidJSONError

The file update_json() was asked to update does not hold valid JSON.

dol.atomic.DFLT_KEEP_HANDLE = True

Whether a LockedAppender keeps its file open between appends. Not on Windows: an open handle there makes the os.replace of a compaction fail.

class dol.atomic.FileLock(path, *, sidecar='.lock', poll_interval=0.001)[source]

Bases: object

A reusable, exclusive, advisory lock on a sidecar file <path><sidecar> (a shortened hidden name instead, for names near the 255-byte limit).

Use it as with lock:; the body runs either way, and lock.held says whether the lock is actually held (it is not when the lock folder is read-only or the filesystem does not support locking: advisory locks are best effort, and the caller decides whether that matters).

The handle is kept open between uses (cheaper for many short sections); close() it when done. It is fork-safe: flock locks belong to the open file description, so a forked child that inherited the handle would not exclude its parent; the pid is remembered and the handle reopened on a mismatch (and a child never releases its parent’s lock). Threads of one process exclude each other too. Not reentrant: entering again from the holding thread raises RuntimeError, and so does entering another FileLock (or file_lock()) of the same path from a thread that already holds it, which would otherwise wait on itself forever. For a one-off section, use file_lock().

The lock file is a separate sidecar, not path itself, so that atomic_write() replacing path does not swap the file being locked. It sits next to the real file (symlinks resolved, as atomic_write() resolves them), so every link to one file shares one lock. The path is resolved each time the lock is entered (a link retargeted while it is held is not followed).

>>> import tempfile
>>> lock = FileLock(os.path.join(tempfile.mkdtemp(), "data.json"))
>>> with lock:
...     lock.held
True
>>> lock.close()
close()[source]

Release the kept handle (not from inside the with block).

Return type:

None

exception dol.atomic.InvalidJSONError[source]

Bases: ValueError

The file update_json() was asked to update does not hold valid JSON.

class dol.atomic.LockedAppender(path, *, lock=None, line_end=None, keep_handle=True)[source]

Bases: object

Appends to path under a FileLock, following the file when an atomic_write() (made under the same lock, e.g. a compaction) replaces it.

Each append is one unbuffered O_APPEND write, so nothing is pending between appends: a failed append leaves nothing to land later, and an inherited handle (after a fork) is safe to share. The handle is kept between appends (keep_handle) and reopened when the file at path is no longer the one it holds, or after a failed append.

To replace the file, hold the appender’s lock, so no append lands in the old file after it was read: with appender.lock: atomic_write(path, compacted). Whether the lock is actually held there is appender.lock.held (see FileLock). with appender: does not lock; it only closes on exit. Atomic appends and tail healing assume the lock is held by every writer: when it cannot be (read-only folder, filesystem without locks), appends are best effort.

>>> import tempfile
>>> path = os.path.join(tempfile.mkdtemp(), "log.jsonl")
>>> log = LockedAppender(path, line_end=b"\n")
>>> log.append(b'{"a": 1}\n')
>>> with log.lock:
...     atomic_write(path, b'{"compacted": true}\n')
>>> log.append(b'{"b": 2}\n')
>>> open(path, "rb").read()
b'{"compacted": true}\n{"b": 2}\n'
>>> log.close()
append(data)[source]

Append data under the lock, to the file now at path.

Return type:

None

close()[source]

Close the file and the lock’s handle (both reopen on the next append). Not from inside with appender.lock: (the lock is not reentrant).

Return type:

None

class dol.atomic.Retry(on=(), attempts=7, backoff_s=0.01)[source]

Bases: object

Call op(*args), retrying on on exceptions with doubling backoff.

attempts calls at most, sleeping backoff_s, then twice that, and so on between them (defaults: 7 calls, ~0.6 s of sleep in all; the backoff is not capped, so keep attempts small). The last failure is re-raised. Any other exception propagates at once. Retry(attempts=1) (or on=()) calls op once.

>>> calls = []
>>> def flaky():
...     calls.append(1)
...     if len(calls) < 3:
...         raise PermissionError("in use")
...     return "done"
>>> Retry(on=(PermissionError,), backoff_s=0)(flaky), len(calls)
('done', 3)
dol.atomic.SHARING_VIOLATIONS: tuple[type[BaseException], ...] = ()

Windows’ sharing violation (another handle has the file open); nothing elsewhere, where PermissionError means a real permission problem. Windows reports a sharing violation on os.replace with the same error as a genuine access denial, so there every PermissionError is retried: a read-only destination still fails, only ~0.6 s later.

Type:

What DFLT_RETRY retries

dol.atomic.TEMP_NAME_PATTERN = re.compile('\\..+\\.[0-9a-f]{8}\\.tmp', re.DOTALL)

Matches (fullmatch) every name temp_name() makes, and so the temp file a killed atomic_write() leaves behind. DOTALL: a file name may hold a newline.

dol.atomic.atomic_delete(path, *, retry=Retry(on=(), attempts=7, backoff_s=0.01))[source]

Remove the file path: retry(os.unlink, path), the delete counterpart of atomic_write().

os.unlink is already atomic; what this adds is the same retry, so a delete is not refused on Windows because a reader has the file open. (Windows may then keep the name “delete pending” until the last handle closes.) A symlink is removed itself, not its target. A missing file raises FileNotFoundError (not retried).

>>> import tempfile
>>> path = os.path.join(tempfile.mkdtemp(), "f")
>>> atomic_write(path, b"x")
>>> atomic_delete(path)
>>> os.path.exists(path)
False
Return type:

None

dol.atomic.atomic_write(path, data, *, fsync=False, perms=None, retry=Retry(on=(), attempts=7, backoff_s=0.01))[source]

Write data to path atomically (temp file in the same folder, then os.replace).

Parameters:
  • path – Target file (str, bytes or path-like); its parent folder must exist. A symlink is followed: its target is what gets replaced.

  • data (bytes | Iterable[bytes]) – Bytes-like, or an iterable of bytes-like chunks (written one at a time, so a large payload need not be held in memory). If the iterable raises, path is untouched. Text must be encoded first.

  • fsync (bool) – Flush the file (and, on POSIX, the folder) to disk before returning: durable across power loss, not just across a process crash. Slower. (On macOS os.fsync is not F_FULLFSYNC: best effort.)

  • perms (int | None) – Mode (e.g. 0o600) for the file, set at creation so the data is never readable more widely, even briefly, and not narrowed by the umask. Default: the mode of the file being replaced, else the usual umask mode.

  • retry (Callable) – How the final os.replace is called: retry(os.replace, tmp, path). Default DFLT_RETRY retries Windows’ sharing violations (a reader has path open); Retry(attempts=1) calls it once.

Return type:

None

The temp file is a hidden, uniquely named sibling (.<name>.<random>.tmp), created with O_EXCL, so concurrent writers never share one and key listings skip it. On any failure it is removed and the old content is untouched. Atomic on POSIX, and on Windows within one volume. Hard links to the old file are detached, and its owner is not preserved.

>>> import tempfile
>>> d = tempfile.mkdtemp()
>>> atomic_write(os.path.join(d, "a"), [b"x", b"y"], perms=0o600, fsync=True)
>>> open(os.path.join(d, "a"), "rb").read()
b'xy'
>>> def boom():
...     yield b"partial"
...     raise RuntimeError("producer died")
>>> atomic_write(os.path.join(d, "a"), boom())
Traceback (most recent call last):
  ...
RuntimeError: producer died
>>> open(os.path.join(d, "a"), "rb").read(), os.listdir(d)  # old content, no litter
(b'xy', ['a'])
dol.atomic.file_lock(path, *, sidecar='.lock', poll_interval=0.001)[source]

Hold an exclusive advisory lock on path’s sidecar (see FileLock); yields whether it is held.

Cross-process and cross-platform (flock on POSIX, msvcrt.locking on Windows). Take it around every append, read and atomic_write() of path so they cannot interleave. The body still runs when locking is unavailable (read-only folder, filesystem without lock support): the yielded value is then False. A lock file that exists but is not writable is opened read-only. Nesting it on the same path in one thread raises RuntimeError (it would deadlock); use one section.

>>> import tempfile
>>> path = os.path.join(tempfile.mkdtemp(), "data.json")
>>> with file_lock(path) as held:
...     atomic_write(path, b"{}")
>>> held, os.path.exists(path + ".lock")
(True, True)
dol.atomic.interrupted_writes(folder, *, pattern=re.compile('\\..+\\.[0-9a-f]{8}\\.tmp', re.DOTALL), recursive=False)[source]

The temp files writes killed mid-way left in folder: sorted absolute paths (str; .. is not collapsed, so each names the file found) of the regular files (never a symlink or folder) whose name fully matches pattern (default: atomic_write()’s temps; pass another writer’s, e.g. r"\.blob-.+\.part").

Matching is by name only: call it on folders your own stores write, where nothing else is named like a temp. A listed file may also be a write still in progress in another process: delete only those older than any write could take (os.stat(p).st_mtime), or while holding the lock every writer takes.

Parameters:

recursive (bool) – Also search subfolders (symlinked folders are not followed; subfolders that cannot be read, or vanish, are skipped).

Return type:

list[str]

A missing folder holds none ([]); any other failure to list it raises OSError.

>>> import tempfile
>>> with tempfile.TemporaryDirectory() as d:
...     for name in ("k.mp4", ".k.mp4.ab12cd34.tmp", ".DS_Store", "x.tmp"):
...         open(os.path.join(d, name), "wb").close()
...     [os.path.basename(p) for p in interrupted_writes(d)]
['.k.mp4.ab12cd34.tmp']
dol.atomic.temp_name(name)[source]

The name atomic_write() stages a write to name under: a hidden sibling .<name, clipped to 100 bytes>.<8 random hex>.tmp, always matched by TEMP_NAME_PATTERN. Unique per call. name is a file name, not a path.

>>> bool(TEMP_NAME_PATTERN.fullmatch(temp_name("data.json")))
True
Return type:

str

dol.atomic.update_json(path, change, *, default=<class 'dict'>, loads=<function loads>, dumps=<function dumps>, lock=None, warn_unlocked=True, write=<function atomic_write>)[source]

Apply change to the JSON document at path under its lock; return what change returned.

Under file_lock() of path: read the document (a dict or list), call change(obj), which mutates obj in place (and may raise, to write nothing), then atomic_write() it back, only if dumps of it changed. So a no-op neither creates the file nor moves its mtime, and a file nobody changed keeps its formatting. What change returns is handed back, never written.

The read is strict: only a missing file starts from default(). Any other read error propagates, and invalid JSON raises InvalidJSONError (a ValueError): writing “default plus this change” over a file that could not be read would destroy what it held.

Parameters:
  • path – The JSON file; its folder must exist.

  • default (Callable | None) – Factory for the document when path does not exist. None means the record must already exist: a missing file raises FileNotFoundError naming path (and, as it is raised before the lock is taken when it can be, creates nothing).

  • dumps (Callable) – The codec. loads gets bytes; dumps may return str or bytes (e.g. functools.partial(json.dumps, indent=2), or orjson’s). Change is detected by comparing dumps before and after, so it must be deterministic.

  • lock –

    The context manager to hold. Default: the FileLock every file_lock() of path takes. Any other works; whether it is held is

    the held of what it yields, else the truth of what it yields (None, e.g. nullcontext(), counts as held). Called while this thread already holds the file’s lock, the default lock raises RuntimeError; inside a section that already holds it, pass lock=contextlib.nullcontext().

  • warn_unlocked (bool) – Warn when the lock is not held (read-only folder, filesystem without locks), so the loss of the guarantee is not silent.

  • write (Callable) – How the new document’s bytes reach path: write(path, data), called under the lock. Default atomic_write() as is; e.g. functools.partial(atomic_write, fsync=True, perms=0o600) for a durable, owner-only record. It must replace path atomically, or a reader can see a torn document.

>>> import tempfile
>>> path = os.path.join(tempfile.mkdtemp(), "manifest.json")
>>> update_json(path, lambda m: m.setdefault("clips", []).append("a.mp4"))
>>> update_json(path, lambda m: len(m["clips"]))
1
>>> open(path).read()
'{"clips": ["a.mp4"]}'
>>> update_json(path, lambda m: {**m, "lost": True})  # returned, not written
{'clips': ['a.mp4'], 'lost': True}
>>> open(path).read()
'{"clips": ["a.mp4"]}'

default=None for a record that must already exist:

>>> update_json(path + ".gone", lambda m: None, default=None)
Traceback (most recent call last):
...
FileNotFoundError: [Errno 2] update_json: the record must exist (default=None): '...manifest.json.gone'
dol.atomic.write_content_addressed(folder, data, *, hasher=<built-in function openssl_sha256>, name_of=<function <lambda>>, fsync=False, perms=None, retry=Retry(on=(), attempts=7, backoff_s=0.01))[source]

Write data into folder under a name computed from its content; return that name (the file is os.path.join(folder, name)).

The atomic_write() of a file whose name is not known until its last byte is: data (bytes-like, or an iterable of chunks, so a large payload streams) is written to a hidden temp in folder (named like atomic_write()’s, so interrupted_writes() finds one a kill left) while being hashed, then the temp is renamed to name_of(hasher). A reader that finds the name finds the whole content, never a prefix. If that name already exists, identical content is already there: the temp is dropped and the existing file left as it is (when it is a regular file of the same size; anything else under that name, a truncated file, a link, is replaced, a folder raises). On any failure the temp is removed.

Parameters:
  • folder – Where the file goes; it must exist.

  • hasher (Callable) – Factory of a fresh hash object (update-able), e.g. hashlib.blake2b or functools.partial(hashlib.sha256, usedforsecurity=False).

  • name_of (Callable) – name_of(hash_object) -> the file name (a plain name, not a path). Default: the hex digest.

  • retry (Callable) – As in atomic_write(). perms defaults to the usual umask mode.

Return type:

str

>>> import tempfile
>>> d = tempfile.mkdtemp()
>>> name = write_content_addressed(d, [b"he", b"llo"])
>>> name[:12], open(os.path.join(d, name), "rb").read(), os.listdir(d) == [name]
('2cf24dba5fb0', b'hello', True)
>>> write_content_addressed(d, b"hello") == name  # same content, same file
True

See also dol.content (a content-addressed store, with a key per content).