dol.atomic¶
Atomic file writes: write to a sibling temp file, then os.replace it into place.
A crash (or an exception while producing the data) leaves the old content or the new,
never a torn or empty file. Dependency-free; atomic_write is the entry point, and
atomic_delete its counterpart for removal, write_content_addressed its variant
for a name computed from the content. A write killed mid-way leaves its temp
file behind; interrupted_writes lists them (TEMP_NAME_PATTERN is their name).
On Windows, a reader holding the destination open makes os.replace and os.unlink
fail with PermissionError until it closes (Python’s open does not share delete).
Both functions therefore retry that briefly, through a Retry policy (its
retry argument): by default Windows’ PermissionError only, ~0.6 s in all.
Elsewhere the same error is a real permission problem, so nothing is retried.
file_lock is its companion: an atomic replace is only safe against concurrent
appenders and readers when they all hold the same advisory lock. LockedAppender
appends under that lock and follows the file when an atomic replace swaps it.
>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), "f.txt")
>>> atomic_write(path, b"hello")
>>> open(path).read(), os.listdir(os.path.dirname(path))
('hello', ['f.txt'])
>>> atomic_write(path, (c.encode() for c in "ab")) # chunks are streamed
>>> open(path).read()
'ab'
Module Attributes
Windows' sharing violation (another handle has the file open); nothing elsewhere, where |
|
Matches ( |
|
Whether a |
Functions
|
Remove the file |
|
Write |
|
Hold an exclusive advisory lock on |
|
The temp files writes killed mid-way left in |
|
The name |
|
Apply |
|
Write |
Classes
|
A reusable, exclusive, advisory lock on a sidecar file |
|
Appends to |
|
Call |
Exceptions
The file |
- dol.atomic.DFLT_KEEP_HANDLE = True¶
Whether a
LockedAppenderkeeps its file open between appends. Not on Windows: an open handle there makes theos.replaceof a compaction fail.
- class dol.atomic.FileLock(path, *, sidecar='.lock', poll_interval=0.001)[source]¶
Bases:
objectA reusable, exclusive, advisory lock on a sidecar file
<path><sidecar>(a shortened hidden name instead, for names near the 255-byte limit).Use it as
with lock:; the body runs either way, andlock.heldsays whether the lock is actually held (it is not when the lock folder is read-only or the filesystem does not support locking: advisory locks are best effort, and the caller decides whether that matters).The handle is kept open between uses (cheaper for many short sections);
close()it when done. It is fork-safe:flocklocks belong to the open file description, so a forked child that inherited the handle would not exclude its parent; the pid is remembered and the handle reopened on a mismatch (and a child never releases its parent’s lock). Threads of one process exclude each other too. Not reentrant: entering again from the holding thread raisesRuntimeError, and so does entering anotherFileLock(orfile_lock()) of the same path from a thread that already holds it, which would otherwise wait on itself forever. For a one-off section, usefile_lock().The lock file is a separate sidecar, not
pathitself, so thatatomic_write()replacingpathdoes not swap the file being locked. It sits next to the real file (symlinks resolved, asatomic_write()resolves them), so every link to one file shares one lock. The path is resolved each time the lock is entered (a link retargeted while it is held is not followed).>>> import tempfile >>> lock = FileLock(os.path.join(tempfile.mkdtemp(), "data.json")) >>> with lock: ... lock.held True >>> lock.close()
- exception dol.atomic.InvalidJSONError[source]¶
Bases:
ValueErrorThe file
update_json()was asked to update does not hold valid JSON.
- class dol.atomic.LockedAppender(path, *, lock=None, line_end=None, keep_handle=True)[source]¶
Bases:
objectAppends to
pathunder aFileLock, following the file when anatomic_write()(made under the same lock, e.g. a compaction) replaces it.Each append is one unbuffered
O_APPENDwrite, so nothing is pending between appends: a failed append leaves nothing to land later, and an inherited handle (after a fork) is safe to share. The handle is kept between appends (keep_handle) and reopened when the file atpathis no longer the one it holds, or after a failed append.To replace the file, hold the appender’s lock, so no append lands in the old file after it was read:
with appender.lock: atomic_write(path, compacted). Whether the lock is actually held there isappender.lock.held(seeFileLock).with appender:does not lock; it only closes on exit. Atomic appends and tail healing assume the lock is held by every writer: when it cannot be (read-only folder, filesystem without locks), appends are best effort.>>> import tempfile >>> path = os.path.join(tempfile.mkdtemp(), "log.jsonl") >>> log = LockedAppender(path, line_end=b"\n") >>> log.append(b'{"a": 1}\n') >>> with log.lock: ... atomic_write(path, b'{"compacted": true}\n') >>> log.append(b'{"b": 2}\n') >>> open(path, "rb").read() b'{"compacted": true}\n{"b": 2}\n' >>> log.close()
- class dol.atomic.Retry(on=(), attempts=7, backoff_s=0.01)[source]¶
Bases:
objectCall
op(*args), retrying ononexceptions with doubling backoff.attemptscalls at most, sleepingbackoff_s, then twice that, and so on between them (defaults: 7 calls, ~0.6 s of sleep in all; the backoff is not capped, so keepattemptssmall). The last failure is re-raised. Any other exception propagates at once.Retry(attempts=1)(oron=()) callsoponce.>>> calls = [] >>> def flaky(): ... calls.append(1) ... if len(calls) < 3: ... raise PermissionError("in use") ... return "done" >>> Retry(on=(PermissionError,), backoff_s=0)(flaky), len(calls) ('done', 3)
- dol.atomic.SHARING_VIOLATIONS: tuple[type[BaseException], ...] = ()¶
Windows’ sharing violation (another handle has the file open); nothing elsewhere, where
PermissionErrormeans a real permission problem. Windows reports a sharing violation onos.replacewith the same error as a genuine access denial, so there everyPermissionErroris retried: a read-only destination still fails, only ~0.6 s later.- Type:
What
DFLT_RETRYretries
- dol.atomic.TEMP_NAME_PATTERN = re.compile('\\..+\\.[0-9a-f]{8}\\.tmp', re.DOTALL)¶
Matches (
fullmatch) every nametemp_name()makes, and so the temp file a killedatomic_write()leaves behind.DOTALL: a file name may hold a newline.
- dol.atomic.atomic_delete(path, *, retry=Retry(on=(), attempts=7, backoff_s=0.01))[source]¶
Remove the file
path:retry(os.unlink, path), the delete counterpart ofatomic_write().os.unlinkis already atomic; what this adds is the same retry, so a delete is not refused on Windows because a reader has the file open. (Windows may then keep the name “delete pending” until the last handle closes.) A symlink is removed itself, not its target. A missing file raisesFileNotFoundError(not retried).>>> import tempfile >>> path = os.path.join(tempfile.mkdtemp(), "f") >>> atomic_write(path, b"x") >>> atomic_delete(path) >>> os.path.exists(path) False
- Return type:
- dol.atomic.atomic_write(path, data, *, fsync=False, perms=None, retry=Retry(on=(), attempts=7, backoff_s=0.01))[source]¶
Write
datatopathatomically (temp file in the same folder, thenos.replace).- Parameters:
path – Target file (
str,bytesor path-like); its parent folder must exist. A symlink is followed: its target is what gets replaced.data (
bytes|Iterable[bytes]) – Bytes-like, or an iterable of bytes-like chunks (written one at a time, so a large payload need not be held in memory). If the iterable raises,pathis untouched. Text must be encoded first.fsync (
bool) – Flush the file (and, on POSIX, the folder) to disk before returning: durable across power loss, not just across a process crash. Slower. (On macOSos.fsyncis notF_FULLFSYNC: best effort.)perms (
int|None) – Mode (e.g.0o600) for the file, set at creation so the data is never readable more widely, even briefly, and not narrowed by the umask. Default: the mode of the file being replaced, else the usual umask mode.retry (
Callable) – How the finalos.replaceis called:retry(os.replace, tmp, path). DefaultDFLT_RETRYretries Windows’ sharing violations (a reader haspathopen);Retry(attempts=1)calls it once.
- Return type:
The temp file is a hidden, uniquely named sibling (
.<name>.<random>.tmp), created withO_EXCL, so concurrent writers never share one and key listings skip it. On any failure it is removed and the old content is untouched. Atomic on POSIX, and on Windows within one volume. Hard links to the old file are detached, and its owner is not preserved.>>> import tempfile >>> d = tempfile.mkdtemp() >>> atomic_write(os.path.join(d, "a"), [b"x", b"y"], perms=0o600, fsync=True) >>> open(os.path.join(d, "a"), "rb").read() b'xy' >>> def boom(): ... yield b"partial" ... raise RuntimeError("producer died") >>> atomic_write(os.path.join(d, "a"), boom()) Traceback (most recent call last): ... RuntimeError: producer died >>> open(os.path.join(d, "a"), "rb").read(), os.listdir(d) # old content, no litter (b'xy', ['a'])
- dol.atomic.file_lock(path, *, sidecar='.lock', poll_interval=0.001)[source]¶
Hold an exclusive advisory lock on
path’s sidecar (seeFileLock); yields whether it is held.Cross-process and cross-platform (
flockon POSIX,msvcrt.lockingon Windows). Take it around every append, read andatomic_write()ofpathso they cannot interleave. The body still runs when locking is unavailable (read-only folder, filesystem without lock support): the yielded value is thenFalse. A lock file that exists but is not writable is opened read-only. Nesting it on the same path in one thread raisesRuntimeError(it would deadlock); use one section.>>> import tempfile >>> path = os.path.join(tempfile.mkdtemp(), "data.json") >>> with file_lock(path) as held: ... atomic_write(path, b"{}") >>> held, os.path.exists(path + ".lock") (True, True)
- dol.atomic.interrupted_writes(folder, *, pattern=re.compile('\\..+\\.[0-9a-f]{8}\\.tmp', re.DOTALL), recursive=False)[source]¶
The temp files writes killed mid-way left in
folder: sorted absolute paths (str;..is not collapsed, so each names the file found) of the regular files (never a symlink or folder) whose name fully matchespattern(default:atomic_write()’s temps; pass another writer’s, e.g.r"\.blob-.+\.part").Matching is by name only: call it on folders your own stores write, where nothing else is named like a temp. A listed file may also be a write still in progress in another process: delete only those older than any write could take (
os.stat(p).st_mtime), or while holding the lock every writer takes.- Parameters:
recursive (
bool) – Also search subfolders (symlinked folders are not followed; subfolders that cannot be read, or vanish, are skipped).- Return type:
A missing
folderholds none ([]); any other failure to list it raisesOSError.>>> import tempfile >>> with tempfile.TemporaryDirectory() as d: ... for name in ("k.mp4", ".k.mp4.ab12cd34.tmp", ".DS_Store", "x.tmp"): ... open(os.path.join(d, name), "wb").close() ... [os.path.basename(p) for p in interrupted_writes(d)] ['.k.mp4.ab12cd34.tmp']
- dol.atomic.temp_name(name)[source]¶
The name
atomic_write()stages a write tonameunder: a hidden sibling.<name, clipped to 100 bytes>.<8 random hex>.tmp, always matched byTEMP_NAME_PATTERN. Unique per call.nameis a file name, not a path.>>> bool(TEMP_NAME_PATTERN.fullmatch(temp_name("data.json"))) True
- Return type:
- dol.atomic.update_json(path, change, *, default=<class 'dict'>, loads=<function loads>, dumps=<function dumps>, lock=None, warn_unlocked=True, write=<function atomic_write>)[source]¶
Apply
changeto the JSON document atpathunder its lock; return whatchangereturned.Under
file_lock()ofpath: read the document (adictorlist), callchange(obj), which mutatesobjin place (and may raise, to write nothing), thenatomic_write()it back, only ifdumpsof it changed. So a no-op neither creates the file nor moves its mtime, and a file nobody changed keeps its formatting. Whatchangereturns is handed back, never written.The read is strict: only a missing file starts from
default(). Any other read error propagates, and invalid JSON raisesInvalidJSONError(aValueError): writing “default plus this change” over a file that could not be read would destroy what it held.- Parameters:
path – The JSON file; its folder must exist.
default (
Callable|None) – Factory for the document whenpathdoes not exist.Nonemeans the record must already exist: a missing file raisesFileNotFoundErrornamingpath(and, as it is raised before the lock is taken when it can be, creates nothing).dumps (
Callable) – The codec.loadsgets bytes;dumpsmay return str or bytes (e.g.functools.partial(json.dumps, indent=2), or orjson’s). Change is detected by comparingdumpsbefore and after, so it must be deterministic.lock –
The context manager to hold. Default: the
FileLockeveryfile_lock()ofpathtakes. Any other works; whether it is held isthe
heldof what it yields, else the truth of what it yields (None, e.g.nullcontext(), counts as held). Called while this thread already holds the file’s lock, the default lock raisesRuntimeError; inside a section that already holds it, passlock=contextlib.nullcontext().warn_unlocked (
bool) – Warn when the lock is not held (read-only folder, filesystem without locks), so the loss of the guarantee is not silent.write (
Callable) – How the new document’s bytes reachpath:write(path, data), called under the lock. Defaultatomic_write()as is; e.g.functools.partial(atomic_write, fsync=True, perms=0o600)for a durable, owner-only record. It must replacepathatomically, or a reader can see a torn document.
>>> import tempfile >>> path = os.path.join(tempfile.mkdtemp(), "manifest.json") >>> update_json(path, lambda m: m.setdefault("clips", []).append("a.mp4")) >>> update_json(path, lambda m: len(m["clips"])) 1 >>> open(path).read() '{"clips": ["a.mp4"]}' >>> update_json(path, lambda m: {**m, "lost": True}) # returned, not written {'clips': ['a.mp4'], 'lost': True} >>> open(path).read() '{"clips": ["a.mp4"]}'
default=Nonefor a record that must already exist:>>> update_json(path + ".gone", lambda m: None, default=None) Traceback (most recent call last): ... FileNotFoundError: [Errno 2] update_json: the record must exist (default=None): '...manifest.json.gone'
- dol.atomic.write_content_addressed(folder, data, *, hasher=<built-in function openssl_sha256>, name_of=<function <lambda>>, fsync=False, perms=None, retry=Retry(on=(), attempts=7, backoff_s=0.01))[source]¶
Write
dataintofolderunder a name computed from its content; return that name (the file isos.path.join(folder, name)).The
atomic_write()of a file whose name is not known until its last byte is:data(bytes-like, or an iterable of chunks, so a large payload streams) is written to a hidden temp infolder(named likeatomic_write()’s, sointerrupted_writes()finds one a kill left) while being hashed, then the temp is renamed toname_of(hasher). A reader that finds the name finds the whole content, never a prefix. If that name already exists, identical content is already there: the temp is dropped and the existing file left as it is (when it is a regular file of the same size; anything else under that name, a truncated file, a link, is replaced, a folder raises). On any failure the temp is removed.- Parameters:
folder – Where the file goes; it must exist.
hasher (
Callable) – Factory of a fresh hash object (update-able), e.g.hashlib.blake2borfunctools.partial(hashlib.sha256, usedforsecurity=False).name_of (
Callable) –name_of(hash_object)-> the file name (a plain name, not a path). Default: the hex digest.retry (
Callable) – As inatomic_write().permsdefaults to the usual umask mode.
- Return type:
>>> import tempfile >>> d = tempfile.mkdtemp() >>> name = write_content_addressed(d, [b"he", b"llo"]) >>> name[:12], open(os.path.join(d, name), "rb").read(), os.listdir(d) == [name] ('2cf24dba5fb0', b'hello', True) >>> write_content_addressed(d, b"hello") == name # same content, same file True
See also
dol.content(a content-addressed store, with a key per content).