dol.filesys

File system access: dict-like stores over folders and files.

Files gives a folder a MutableMapping interface: keys are paths relative to the root folder, values are the files’ bytes. TextFiles, JsonFiles and PickleFiles add the corresponding value codecs. Writing under a sub-folder that does not exist raises KeyError; wrap the store with mk_dirs_if_missing to create folders on write.

Main entry points:

  • Files: bytes of the files under a root folder

  • TextFiles: same, with text values

  • JsonFiles: same, with JSON-decoded values

  • PickleFiles: same, with pickled values

  • mk_dirs_if_missing: make a file store create missing directories on write

  • dir_made: make the directories for a write, unmake them if it fails

    >>> import tempfile
    >>> s = Files(tempfile.mkdtemp())
    >>> s['hello.txt'] = b'world'
    >>> s['hello.txt']
    b'world'
    >>> list(s)
    ['hello.txt']
    

Functions

create_directories(dirpath[, max_dirs_to_make])

Create directories up to a specified limit.

dir_made(directory, *[, verbose])

Context manager: make directory (and its missing ancestors) for a write.

dotdot_resolves_lexically(path, *[, base])

Whether every .. in path undoes the segment before it, as os.path.normpath assumes.

ensure_dir(dirpath, *[, max_dirs_to_make, ...])

Ensure that a directory exists, creating it if necessary.

ensure_slash_suffix(path)

Add a file separation (/ or ) at the end of path str, if not already present.

iter_dirpaths_in_folder_recursively(root_folder)

Recursively generates dirpaths of folder (and subfolders, etc.) up to a given level

iter_filepaths_in_folder_recursively(root_folder)

Recursively generates filepaths of folder (and subfolders, etc.) up to a given level

mk_absolute_path(path_format)

Expand a leading ~, or make a leading . path absolute; other paths are returned as is.

mk_dirs_if_missing([store_cls, ...])

Store decorator that will make the store create directories on write as needed.

mk_dirs_if_missing_preset(self, k, v, *[, ...])

Preset function that will make the store create directories on write as needed.

mk_json_bytes_wrap(*[, loads_kwargs, ...])

Make a wrap_kvs value-codec wrapper for JSON, with kwargs for json.loads/json.dumps.

mk_pickle_bytes_wrap(*[, loads_kwargs, ...])

Make a wrap_kvs value-codec wrapper for pickle, with kwargs for pickle.loads/pickle.dumps.

mk_tmp_dol_dir([dirname, ...])

Create and return a path to a temporary directory that's guaranteed to be accessible to the user.

path_is_under(path, rootdir)

Whether path, lexically normalized, is rootdir or lies under it.

path_of(store, key)

The file-system path of key in a file-backed store, however it is wrapped.

paths_in_dir(rootdir[, include_hidden])

Yield the paths of the entries of rootdir (directories with a trailing separator), skipping hidden ones unless include_hidden.

process_path(*path[, ensure_dir_exists, ...])

Process a path string, ensuring it exists, and optionally expanding user.

resolve_dir(dirpath[, assert_existence, ...])

Resolve a path to a full, real, path to a directory

resolve_path(path[, assert_existence])

Resolve a path to a full, real, (file or folder) path (opt assert existence).

subfolder_stores(root_folder, *[, ...])

Create a store of subfolders of a given folder, where the keys are the subfolder paths (by default, relative and slash-less) and the values are stores of these subfolders.

temp_dir([dirname, make_it_if_necessary, ...])

Create and return a path to a temporary directory that's guaranteed to be accessible to the user.

validate_key_and_raise_key_error_on_exception(func)

Method decorator: validate the key first, and re-raise any exception of the method as a KeyError.

Classes

DirCollection(rootdir[, subpath, ...])

Collection of the directory paths under rootdir.

DirReader(rootdir[, subpath, ...])

Reader mapping each sub-directory of rootdir to a DirReader of it.

FileBytesPersister(*args[, delete_func, ...])

File persistence with configurable deletion.

FileBytesReader(*args[, read_retry, ...])

Reader mapping file paths under rootdir to the files' bytes.

FileCollection(rootdir[, subpath, ...])

Collection of the file paths under rootdir.

FileInfoReader(rootdir[, subpath, ...])

Reader mapping file paths to their os.stat result.

FileStringPersister(*args[, delete_func, ...])

Persister mapping file paths to the files' text (files opened in text mode).

FileStringReader(*args[, read_retry, ...])

Reader mapping file paths to the files' text (files opened in text mode).

FileSysCollection(rootdir[, subpath, ...])

Base collection of file-system paths under rootdir, optionally restricted by subpath, max_levels and hidden-file inclusion.

Files(*args[, delete_func, ...])

FileBytesPersister with relative paths

FilesReader(*args[, read_retry, ...])

FileBytesReader with relative paths

JsonFiles(*args[, delete_func, ...])

A store of json files

Jsons(*args[, delete_func, ...])

Like JsonFiles, but with added .json extension handling Namely: filtering for .json extensions but not showing the extension in keys

LocalFileDeleteMixin()

Mixin providing configurable file deletion.

MakeMissingDirsStoreMixin()

Will make a local file store automatically create the directories needed to create a file.

PickleFiles(*args[, delete_func, ...])

A store of pickles

PickleStore

PickleStores(rootdir[, subpath, ...])

Reader mapping each sub-directory of rootdir to a PickleFiles store of it.

RelPathFileBytesPersister

RelPathFileBytesReader

RelPathFileStringPersister

RelPathFileStringReader

ReprMixin()

A __repr__ showing the _init_kwargs the instance was created with.

TextFiles(*args[, delete_func, ...])

FileStringPersister with relative paths

TextFilesReader(*args[, read_retry, ...])

FileStringReader with relative paths

Exceptions

KeyValidationError

A KeyError for keys that fail a file-system store's validation.

class dol.filesys.DirCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]

Bases: FileSysCollection

Collection of the directory paths under rootdir.

class dol.filesys.DirReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]

Bases: DirCollection, KvReader

Reader mapping each sub-directory of rootdir to a DirReader of it.

class dol.filesys.FileBytesPersister(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]

Bases: LocalFileDeleteMixin, FileBytesReader, KvPersister

File persistence with configurable deletion.

Supports custom deletion functions via delete_func parameter in __init__.

By default, tries to move files to trash with fallback to os.remove. See dol.trash module for deletion strategies: permanent_delete, trash_only, etc.

class dol.filesys.FileBytesReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]

Bases: FileCollection, KvReader

Reader mapping file paths under rootdir to the files’ bytes.

class dol.filesys.FileCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]

Bases: FileSysCollection

Collection of the file paths under rootdir.

class dol.filesys.FileInfoReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]

Bases: FileCollection, KvReader

Reader mapping file paths to their os.stat result.

class dol.filesys.FileStringPersister(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]

Bases: FileBytesPersister

Persister mapping file paths to the files’ text (files opened in text mode).

Reads and writes as UTF-8 explicitly, rather than inheriting locale.getpreferredencoding() (see i2mint/dol#97): a store is a serialization boundary, and one whose format silently depends on an ambient environment variable isn’t really specified. Without this, a write can raise on a non-ASCII-locale machine, or a store synced between two machines with different locales can silently corrupt on round trip.

class dol.filesys.FileStringReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]

Bases: FileBytesReader

Reader mapping file paths to the files’ text (files opened in text mode).

Reads as UTF-8 explicitly, rather than inheriting locale.getpreferredencoding() (see i2mint/dol#97): a store is a serialization boundary, and one whose format silently depends on an ambient environment variable isn’t really specified. This also matches how FileStringPersister writes (below), so a round trip is safe regardless of which locale reads or writes.

class dol.filesys.FileSysCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]

Bases: Collection

Base collection of file-system paths under rootdir, optionally restricted by subpath, max_levels and hidden-file inclusion.

Keys that would leave rootdir through .. segments (e.g. '<rootdir>/../secret') are not valid keys: reading, writing or deleting them raises KeyError, and in returns False (i2mint/dol#107). Pass the keyword-only allow_escape=True to address paths outside the root on purpose.

is_in_root(k)[source]

Whether k stays under rootdir (always true with allow_escape=True or an empty rootdir).

The containment test is lexical (path_is_under()), so a key whose .. follows a symlink – where the OS climbs out of the link’s target, not back to the link’s directory – is refused too (i2mint/dol#123).

is_valid_key(k)[source]

Whether k is a key of this collection: it must match the key pattern and, unless allow_escape=True, stay under rootdir once .. segments are resolved (see path_is_under()). An empty rootdir (as in Files(''), used by dol.misc) confines nothing: its keys are arbitrary paths by design.

validate_in_root(k)[source]

Raise KeyValidationError if k leaves rootdir (i2mint/dol#107).

For methods that build a value from k without a full validate_key.

with_relative_paths()[source]

Return a copy of self with relative paths

class dol.filesys.Files(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]

Bases: PrefixRelativizationMixin, Store

FileBytesPersister with relative paths

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

class dol.filesys.FilesReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]

Bases: PrefixRelativizationMixin, Store

FileBytesReader with relative paths

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

class dol.filesys.JsonFiles(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]

Bases: Store

A store of json files

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

class dol.filesys.Jsons(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]

Bases: Store

Like JsonFiles, but with added .json extension handling Namely: filtering for .json extensions but not showing the extension in keys

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

exception dol.filesys.KeyValidationError[source]

Bases: KeyError

A KeyError for keys that fail a file-system store’s validation.

class dol.filesys.LocalFileDeleteMixin[source]

Bases: object

Mixin providing configurable file deletion.

The deletion function can be configured either at class level by setting the _delete_func class attribute, or at instance level by setting the _delete_func instance attribute.

By default, uses safe deletion that tries to move to trash with fallback to os.remove (with warning).

See dol.trash module for available deletion strategies:

  • default_delete_func: Safe trash with warning on fallback

  • permanent_delete: Direct os.remove (no warnings)

  • trash_only: Error if trash unavailable

class dol.filesys.MakeMissingDirsStoreMixin[source]

Bases: object

Will make a local file store automatically create the directories needed to create a file. Should be placed before the concrete perisister in the mro but in such a manner so that it receives full paths.

class dol.filesys.PickleFiles(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]

Bases: Store

A store of pickles

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

dol.filesys.PickleStore

alias of PickleFiles

class dol.filesys.PickleStores(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]

Bases: PrefixRelativizationMixin, Store

Reader mapping each sub-directory of rootdir to a PickleFiles store of it.

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

dol.filesys.RelPathFileBytesPersister

alias of Files

dol.filesys.RelPathFileBytesReader

alias of FilesReader

dol.filesys.RelPathFileStringPersister

alias of TextFiles

dol.filesys.RelPathFileStringReader

alias of TextFilesReader

class dol.filesys.ReprMixin[source]

Bases: object

A __repr__ showing the _init_kwargs the instance was created with.

class dol.filesys.TextFiles(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]

Bases: PrefixRelativizationMixin, Store

FileStringPersister with relative paths

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

class dol.filesys.TextFilesReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]

Bases: PrefixRelativizationMixin, Store

FileStringReader with relative paths

is_valid_key(k, *args, __name='is_valid_key', **kwargs)

is_valid_key on the inner key – see mk_relative_path_store.

validate_key(k, *args, __name='validate_key', **kwargs)

validate_key on the inner key – see mk_relative_path_store.

dol.filesys.create_directories(dirpath, max_dirs_to_make=None)[source]

Create directories up to a specified limit.

Parameters:
  • dirpath – The directory path to create.

  • max_dirs_to_make (int | None) – The maximum number of directories to create. If None, there’s no limit.

Returns:

True if the directory exists (already, or after creation); False if creating it would need more than max_dirs_to_make new directories (none are made).

Raises:

ValueError – If max_dirs_to_make is negative.

Examples

>>> import tempfile, shutil
>>> temp_dir = tempfile.mkdtemp()
>>> target_dir = os.path.join(temp_dir, 'a', 'b', 'c')
>>> create_directories(target_dir, max_dirs_to_make=2)
False
>>> create_directories(target_dir, max_dirs_to_make=3)
True
>>> os.path.isdir(target_dir)
True
>>> shutil.rmtree(temp_dir)  # Cleanup
>>> temp_dir = tempfile.mkdtemp()
>>> target_dir = os.path.join(temp_dir, 'a', 'b', 'c', 'd')
>>> create_directories(target_dir)
True
>>> os.path.isdir(target_dir)
True
>>> shutil.rmtree(temp_dir)  # Cleanup
dol.filesys.dir_made(directory, *, verbose=False)[source]

Context manager: make directory (and its missing ancestors) for a write.

If the with body (or the making) raises, the directories this call made are removed again, deepest first, provided they are still empty, so a failed write leaves no stray folder behind. A folder another writer made meanwhile is not this call’s and stays. Use it for writes that do not go through a Files store (temp-file spools, atomic_write()). Yields the directories made, top-down, as a tuple of str.

>>> import tempfile
>>> with tempfile.TemporaryDirectory() as root:
...     target = os.path.join(root, "a", "b")
...     try:
...         with dir_made(target):
...             raise RuntimeError("the write failed")
...     except RuntimeError:
...         pass
...     os.listdir(root)
[]
>>> with tempfile.TemporaryDirectory() as root:
...     with dir_made(os.path.join(root, "a", "b")) as made:
...         len(made)
2
dol.filesys.dotdot_resolves_lexically(path, *, base='')[source]

Whether every .. in path undoes the segment before it, as os.path.normpath assumes.

It doesn’t when a symlink comes before a ..: the OS then climbs out of the link’s target, so root/lnk/../x with root/lnk -> /elsewhere/deep addresses /elsewhere/x, not root/x (i2mint/dol#123). Paths without a .. segment are not touched (no file-system access); otherwise the part up to the last .. is resolved both ways and compared.

With base (a string prefix of path, e.g. a store’s rootdir), only the .. segments after it count: base itself is taken as the OS resolves it, whatever .. it holds. Conservative by design: a .. after a symlink is refused even when the link points back inside base.

>>> dotdot_resolves_lexically('/data/root/a/b.txt')
True
>>> dotdot_resolves_lexically('/no/such/dir/../file')  # nothing to follow: lexical
True
Return type:

bool

dol.filesys.ensure_dir(dirpath, *, max_dirs_to_make=None, verbose=False)[source]

Ensure that a directory exists, creating it if necessary.

Parameters:
  • dirpath – path to the directory to create

  • max_dirs_to_make (int | None) – the maximum number of directories to create. If None, there’s no limit.

  • verbose (bool | str | Callable) – controls verbosity (the noise ensure_dir makes if it make folder)

Returns:

the path to the directory

When the path does not exist, if verbose is:

  • a bool’ a standard message will be printed

  • a callable; will be called on dirpath before directory is created – you can use this to ask the user for confirmation for example

  • a ‘’string``; this string will be printed

Usage note: If you want to string or the (argument-less) callable to be dependent on dirpath, you need make them so when calling ensure_dir.

dol.filesys.ensure_slash_suffix(path)[source]

Add a file separation (/ or ) at the end of path str, if not already present.

An empty path stays empty: an empty prefix has no slash to “ensure”, and turning it into a bare separator anchors otherwise-absolute keys to the filesystem root. On Windows that produces invalid paths like \C:\Users\... (a separator before the drive letter -> OSError: [Errno 22]); e.g. Files("") used with absolute keys, as in dol.misc.get_obj.

dol.filesys.iter_dirpaths_in_folder_recursively(root_folder, max_levels=None, _current_level=0, include_hidden=False)[source]

Recursively generates dirpaths of folder (and subfolders, etc.) up to a given level

dol.filesys.iter_filepaths_in_folder_recursively(root_folder, max_levels=None, _current_level=0, include_hidden=False)[source]

Recursively generates filepaths of folder (and subfolders, etc.) up to a given level

dol.filesys.mk_absolute_path(path_format)[source]

Expand a leading ~, or make a leading . path absolute; other paths are returned as is.

dol.filesys.mk_dirs_if_missing(store_cls=None, *, max_dirs_to_make=None, verbose=False, key_condition=None, __module__=None, __name__=None, __qualname__=None, __doc__=None, __annotations__=None, __defaults__=None, __kwdefaults__=None)[source]

Store decorator that will make the store create directories on write as needed.

Note that it’ll only effect paths relative to the rootdir, which needs to be ensured to exist separatedly.

The directories are made just before the write. If the write then fails (a layer below refuses the key, e.g. a filt_iter scope, or the write itself errors), the directories this call made are removed again (those still empty), so a rejected key leaves no stray folder behind (i2mint/dol#122). (A concurrent writer that had just seen such a directory exist can still lose it to this cleanup: its write then fails like any write to a missing directory.)

dol.filesys.mk_dirs_if_missing_preset(self, k, v, *, max_dirs_to_make=None, verbose=False)[source]

Preset function that will make the store create directories on write as needed.

It only makes the directory: the write itself is left to the wrapper that calls this preset, so it happens exactly once and its errors propagate (#112). The directory is made only when the innermost layer is a file-system store, the key resolves (through every layer’s _id_of_key) to a path whose directory is missing, and that layer’s validate_key accepts it – so an invalid key (e.g. one leaving rootdir, #107) never gets a directory, and the write raises for it.

(mk_dirs_if_missing itself no longer goes through this preset: it also has to see the write fail, to remove the directories it made for a key a layer below rejects, i2mint/dol#122.)

dol.filesys.mk_json_bytes_wrap(*, loads_kwargs=None, dumps_kwargs=None)[source]

Make a wrap_kvs value-codec wrapper for JSON, with kwargs for json.loads/json.dumps.

Return type:

Callable

dol.filesys.mk_pickle_bytes_wrap(*, loads_kwargs=None, dumps_kwargs=None)[source]

Make a wrap_kvs value-codec wrapper for pickle, with kwargs for pickle.loads/pickle.dumps.

Return type:

Callable

dol.filesys.mk_tmp_dol_dir(dirname='', make_it_if_necessary=True, verbose=False)

Create and return a path to a temporary directory that’s guaranteed to be accessible to the user.

Parameters:
  • dirname – Optional subdirectory name to append to the temporary directory path

  • make_it_if_necessary – Whether to create the directory if it doesn’t exist

  • verbose – Controls verbosity when creating directories

Returns:

Path to a temporary directory that the user has access to

Note

This function creates a user-specific temporary directory to avoid permission issues with system-wide temporary directories.

dol.filesys.path_is_under(path, rootdir)[source]

Whether path, lexically normalized, is rootdir or lies under it.

A startswith check is not enough: '/data/root/../secret' starts with '/data/root/' but addresses /data/secret (i2mint/dol#107). The check is lexical (os.path.normpath, not realpath), so a symlinked sub-directory of the root still counts as inside it.

>>> path_is_under('/data/root/a/b.txt', '/data/root/')
True
>>> path_is_under('/data/root/a/../b.txt', '/data/root/')
True
>>> path_is_under('/data/root/../secret.txt', '/data/root/')
False
>>> path_is_under('/data/root/a/../../secret.txt', '/data/root')
False
>>> path_is_under('/data/rootsibling/x', '/data/root')
False
>>> path_is_under('//data/root/a', '//data/root/')
True
>>> path_is_under('root/../a', 'root/..'), path_is_under('root/../../a', 'root/..')
(True, False)
Return type:

bool

dol.filesys.path_of(store, key)[source]

The file-system path of key in a file-backed store, however it is wrapped.

The key is followed through every layer’s _id_of_key (the same walk as dol.inner_most_key) down to the innermost file-system store, which must accept the result: a key leaving rootdir raises KeyValidationError (a KeyError), as a read or write of it would. A store with no file-system layer at its core, or with a layer that maps keys without defining _id_of_key (so the key cannot be followed), raises TypeError. Nothing is read or created. The result is an absolute, normalized path (relative roots resolve against the CWD).

Limit: like dol.dig.key_in_scope, this reads what the layers declare through _id_of_key; a layer that maps keys by other means (add_path_access, flatten) is not followed, and a filt_iter scope is not enforced.

>>> import tempfile
>>> from dol import Files, KeyCodecs
>>> root = tempfile.mkdtemp()
>>> path_of(Files(root), os.path.join('a', 'b.txt')) == os.path.join(root, 'a', 'b.txt')
True
>>> s = KeyCodecs.suffixed('.json')(Files(root))  # wrapped: the outer key is mapped
>>> path_of(s, 'a').endswith('a.json')
True
>>> path_of(Files(root), '../x')
Traceback (most recent call last):
  ...
dol.filesys.KeyValidationError: 'Key not valid (usually because does not exist or access not permitted): ...'
>>> path_of({}, 'a')
Traceback (most recent call last):
  ...
TypeError: ...
Return type:

str

dol.filesys.paths_in_dir(rootdir, include_hidden=False)[source]

Yield the paths of the entries of rootdir (directories with a trailing separator), skipping hidden ones unless include_hidden.

dol.filesys.process_path(*path, ensure_dir_exists=False, assert_exists=False, ensure_endswith_slash=False, ensure_does_not_end_with_slash=False, expanduser=True, expandvars=True, abspath=True, rootdir='')[source]

Process a path string, ensuring it exists, and optionally expanding user.

Parameters:
  • path (Iterable[str]) – The path to process. Can be multiple components of a path.

  • ensure_dir_exists (int | bool) – Whether to ensure the path exists.

  • assert_exists (bool) – Whether to assert that the path exists.

  • ensure_endswith_slash (bool) – Whether to ensure the path ends with a slash.

  • ensure_does_not_end_with_slash (bool) – Whether to ensure the path does not end with a slash.

  • expanduser (bool) – Whether to expand the user in the path.

  • expandvars (bool) – Whether to expand environment variables in the path.

  • abspath (bool) – Whether to convert the path to an absolute path.

  • rootdir (str) – The root directory to prepend to the path.

Returns:

The processed path.

Return type:

str

The result uses the running OS’s native separator, so these examples assert OS-independently (the literal forward-slash form is what you get on POSIX):

>>> import os
>>> process_path('a', 'b', 'c').endswith(os.path.join('a', 'b', 'c'))
True
>>> p = process_path(
...     'a', 'b', 'c', rootdir='root_dir',
...     ensure_endswith_slash=True, abspath=False, expanduser=False, expandvars=False,
... )
>>> p == os.path.join('root_dir', 'a', 'b', 'c') + os.sep
True
dol.filesys.resolve_dir(dirpath, assert_existence=False, ensure_existence=False)[source]

Resolve a path to a full, real, path to a directory

dol.filesys.resolve_path(path, assert_existence=False)[source]

Resolve a path to a full, real, (file or folder) path (opt assert existence). That is, resolve situations where ~ and . prefix the paths.

dol.filesys.subfolder_stores(root_folder, *, max_levels=None, include_hidden=False, relative_paths=True, slash_suffix=False, folder_to_store=<class 'dol.filesys.Files'>)[source]

Create a store of subfolders of a given folder, where the keys are the subfolder paths (by default, relative and slash-less) and the values are stores of these subfolders.

By default, all subfolders will be taken, recursively, but this can be controlled by the max_levels parameter.

dol.filesys.temp_dir(dirname='', make_it_if_necessary=True, verbose=False)[source]

Create and return a path to a temporary directory that’s guaranteed to be accessible to the user.

Parameters:
  • dirname – Optional subdirectory name to append to the temporary directory path

  • make_it_if_necessary – Whether to create the directory if it doesn’t exist

  • verbose – Controls verbosity when creating directories

Returns:

Path to a temporary directory that the user has access to

Note

This function creates a user-specific temporary directory to avoid permission issues with system-wide temporary directories.

dol.filesys.validate_key_and_raise_key_error_on_exception(func)[source]

Method decorator: validate the key first, and re-raise any exception of the method as a KeyError.

Which exceptions become a KeyError is the store’s _reads_as_missing(error, k) (default: every Exception; see FileBytesReader(strict_key_errors=...)).