dol.filesys¶
File system access: dict-like stores over folders and files.
Files gives a folder a MutableMapping interface: keys are paths relative to the
root folder, values are the files’ bytes. TextFiles, JsonFiles and PickleFiles
add the corresponding value codecs. Writing under a sub-folder that does not exist raises
KeyError; wrap the store with mk_dirs_if_missing to create folders on write.
Main entry points:
Files: bytes of the files under a root folderTextFiles: same, with text valuesJsonFiles: same, with JSON-decoded valuesPickleFiles: same, with pickled valuesmk_dirs_if_missing: make a file store create missing directories on writedir_made: make the directories for a write, unmake them if it fails>>> import tempfile >>> s = Files(tempfile.mkdtemp()) >>> s['hello.txt'] = b'world' >>> s['hello.txt'] b'world' >>> list(s) ['hello.txt']
Functions
|
Create directories up to a specified limit. |
|
Context manager: make |
|
Whether every |
|
Ensure that a directory exists, creating it if necessary. |
|
Add a file separation (/ or ) at the end of path str, if not already present. |
|
Recursively generates dirpaths of folder (and subfolders, etc.) up to a given level |
|
Recursively generates filepaths of folder (and subfolders, etc.) up to a given level |
|
Expand a leading |
|
Store decorator that will make the store create directories on write as needed. |
|
Preset function that will make the store create directories on write as needed. |
|
Make a |
|
Make a |
|
Create and return a path to a temporary directory that's guaranteed to be accessible to the user. |
|
Whether |
|
The file-system path of |
|
Yield the paths of the entries of |
|
Process a path string, ensuring it exists, and optionally expanding user. |
|
Resolve a path to a full, real, path to a directory |
|
Resolve a path to a full, real, (file or folder) path (opt assert existence). |
|
Create a store of subfolders of a given folder, where the keys are the subfolder paths (by default, relative and slash-less) and the values are stores of these subfolders. |
|
Create and return a path to a temporary directory that's guaranteed to be accessible to the user. |
Method decorator: validate the key first, and re-raise any exception of the method as a |
Classes
|
Collection of the directory paths under |
|
Reader mapping each sub-directory of |
|
File persistence with configurable deletion. |
|
Reader mapping file paths under |
|
Collection of the file paths under |
|
Reader mapping file paths to their |
|
Persister mapping file paths to the files' text (files opened in text mode). |
|
Reader mapping file paths to the files' text (files opened in text mode). |
|
Base collection of file-system paths under |
|
FileBytesPersister with relative paths |
|
FileBytesReader with relative paths |
|
A store of json files |
|
Like JsonFiles, but with added .json extension handling Namely: filtering for |
Mixin providing configurable file deletion. |
|
Will make a local file store automatically create the directories needed to create a file. |
|
|
A store of pickles |
|
Reader mapping each sub-directory of |
A |
|
|
FileStringPersister with relative paths |
|
FileStringReader with relative paths |
Exceptions
A |
- class dol.filesys.DirCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]¶
Bases:
FileSysCollectionCollection of the directory paths under
rootdir.
- class dol.filesys.DirReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]¶
Bases:
DirCollection,KvReaderReader mapping each sub-directory of
rootdirto aDirReaderof it.
- class dol.filesys.FileBytesPersister(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]¶
Bases:
LocalFileDeleteMixin,FileBytesReader,KvPersisterFile persistence with configurable deletion.
Supports custom deletion functions via delete_func parameter in __init__.
By default, tries to move files to trash with fallback to os.remove. See dol.trash module for deletion strategies: permanent_delete, trash_only, etc.
- class dol.filesys.FileBytesReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]¶
Bases:
FileCollection,KvReaderReader mapping file paths under
rootdirto the files’ bytes.
- class dol.filesys.FileCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]¶
Bases:
FileSysCollectionCollection of the file paths under
rootdir.
- class dol.filesys.FileInfoReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]¶
Bases:
FileCollection,KvReaderReader mapping file paths to their
os.statresult.
- class dol.filesys.FileStringPersister(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]¶
Bases:
FileBytesPersisterPersister mapping file paths to the files’ text (files opened in text mode).
Reads and writes as UTF-8 explicitly, rather than inheriting
locale.getpreferredencoding()(see i2mint/dol#97): a store is a serialization boundary, and one whose format silently depends on an ambient environment variable isn’t really specified. Without this, a write can raise on a non-ASCII-locale machine, or a store synced between two machines with different locales can silently corrupt on round trip.
- class dol.filesys.FileStringReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]¶
Bases:
FileBytesReaderReader mapping file paths to the files’ text (files opened in text mode).
Reads as UTF-8 explicitly, rather than inheriting
locale.getpreferredencoding()(see i2mint/dol#97): a store is a serialization boundary, and one whose format silently depends on an ambient environment variable isn’t really specified. This also matches howFileStringPersisterwrites (below), so a round trip is safe regardless of which locale reads or writes.
- class dol.filesys.FileSysCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]¶
Bases:
CollectionBase collection of file-system paths under
rootdir, optionally restricted bysubpath,max_levelsand hidden-file inclusion.Keys that would leave
rootdirthrough..segments (e.g.'<rootdir>/../secret') are not valid keys: reading, writing or deleting them raisesKeyError, andinreturnsFalse(i2mint/dol#107). Pass the keyword-onlyallow_escape=Trueto address paths outside the root on purpose.- is_in_root(k)[source]¶
Whether
kstays underrootdir(always true withallow_escape=Trueor an emptyrootdir).The containment test is lexical (
path_is_under()), so a key whose..follows a symlink – where the OS climbs out of the link’s target, not back to the link’s directory – is refused too (i2mint/dol#123).
- is_valid_key(k)[source]¶
Whether
kis a key of this collection: it must match the key pattern and, unlessallow_escape=True, stay underrootdironce..segments are resolved (seepath_is_under()). An emptyrootdir(as inFiles(''), used bydol.misc) confines nothing: its keys are arbitrary paths by design.
- class dol.filesys.Files(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]¶
Bases:
PrefixRelativizationMixin,StoreFileBytesPersister with relative paths
- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- class dol.filesys.FilesReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]¶
Bases:
PrefixRelativizationMixin,StoreFileBytesReader with relative paths
- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- class dol.filesys.JsonFiles(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]¶
Bases:
StoreA store of json files
- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- class dol.filesys.Jsons(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]¶
Bases:
StoreLike JsonFiles, but with added .json extension handling Namely: filtering for
.jsonextensions but not showing the extension in keys- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- exception dol.filesys.KeyValidationError[source]¶
Bases:
KeyErrorA
KeyErrorfor keys that fail a file-system store’s validation.
- class dol.filesys.LocalFileDeleteMixin[source]¶
Bases:
objectMixin providing configurable file deletion.
The deletion function can be configured either at class level by setting the _delete_func class attribute, or at instance level by setting the _delete_func instance attribute.
By default, uses safe deletion that tries to move to trash with fallback to os.remove (with warning).
See dol.trash module for available deletion strategies:
default_delete_func: Safe trash with warning on fallback
permanent_delete: Direct os.remove (no warnings)
trash_only: Error if trash unavailable
- class dol.filesys.MakeMissingDirsStoreMixin[source]¶
Bases:
objectWill make a local file store automatically create the directories needed to create a file. Should be placed before the concrete perisister in the mro but in such a manner so that it receives full paths.
- class dol.filesys.PickleFiles(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]¶
Bases:
StoreA store of pickles
- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- dol.filesys.PickleStore¶
alias of
PickleFiles
- class dol.filesys.PickleStores(rootdir, subpath='', pattern_for_field=None, max_levels=None, *, include_hidden=False, assert_rootdir_existence=False, allow_escape=False)[source]¶
Bases:
PrefixRelativizationMixin,StoreReader mapping each sub-directory of
rootdirto aPickleFilesstore of it.- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- dol.filesys.RelPathFileBytesReader¶
alias of
FilesReader
- dol.filesys.RelPathFileStringReader¶
alias of
TextFilesReader
- class dol.filesys.ReprMixin[source]¶
Bases:
objectA
__repr__showing the_init_kwargsthe instance was created with.
- class dol.filesys.TextFiles(*args, delete_func=None, write_open_kwargs=None, write_func=None, **kwargs)[source]¶
Bases:
PrefixRelativizationMixin,StoreFileStringPersister with relative paths
- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- class dol.filesys.TextFilesReader(*args, read_retry=None, strict_key_errors=False, **kwargs)[source]¶
Bases:
PrefixRelativizationMixin,StoreFileStringReader with relative paths
- is_valid_key(k, *args, __name='is_valid_key', **kwargs)¶
is_valid_keyon the inner key – seemk_relative_path_store.
- validate_key(k, *args, __name='validate_key', **kwargs)¶
validate_keyon the inner key – seemk_relative_path_store.
- dol.filesys.create_directories(dirpath, max_dirs_to_make=None)[source]¶
Create directories up to a specified limit.
- Parameters:
- Returns:
True if the directory exists (already, or after creation); False if creating it would need more than
max_dirs_to_makenew directories (none are made).- Raises:
ValueError – If max_dirs_to_make is negative.
Examples
>>> import tempfile, shutil >>> temp_dir = tempfile.mkdtemp() >>> target_dir = os.path.join(temp_dir, 'a', 'b', 'c') >>> create_directories(target_dir, max_dirs_to_make=2) False >>> create_directories(target_dir, max_dirs_to_make=3) True >>> os.path.isdir(target_dir) True >>> shutil.rmtree(temp_dir) # Cleanup
>>> temp_dir = tempfile.mkdtemp() >>> target_dir = os.path.join(temp_dir, 'a', 'b', 'c', 'd') >>> create_directories(target_dir) True >>> os.path.isdir(target_dir) True >>> shutil.rmtree(temp_dir) # Cleanup
- dol.filesys.dir_made(directory, *, verbose=False)[source]¶
Context manager: make
directory(and its missing ancestors) for a write.If the
withbody (or the making) raises, the directories this call made are removed again, deepest first, provided they are still empty, so a failed write leaves no stray folder behind. A folder another writer made meanwhile is not this call’s and stays. Use it for writes that do not go through aFilesstore (temp-file spools,atomic_write()). Yields the directories made, top-down, as a tuple ofstr.>>> import tempfile >>> with tempfile.TemporaryDirectory() as root: ... target = os.path.join(root, "a", "b") ... try: ... with dir_made(target): ... raise RuntimeError("the write failed") ... except RuntimeError: ... pass ... os.listdir(root) [] >>> with tempfile.TemporaryDirectory() as root: ... with dir_made(os.path.join(root, "a", "b")) as made: ... len(made) 2
- dol.filesys.dotdot_resolves_lexically(path, *, base='')[source]¶
Whether every
..inpathundoes the segment before it, asos.path.normpathassumes.It doesn’t when a symlink comes before a
..: the OS then climbs out of the link’s target, soroot/lnk/../xwithroot/lnk -> /elsewhere/deepaddresses/elsewhere/x, notroot/x(i2mint/dol#123). Paths without a..segment are not touched (no file-system access); otherwise the part up to the last..is resolved both ways and compared.With
base(a string prefix ofpath, e.g. a store’srootdir), only the..segments after it count:baseitself is taken as the OS resolves it, whatever..it holds. Conservative by design: a..after a symlink is refused even when the link points back insidebase.>>> dotdot_resolves_lexically('/data/root/a/b.txt') True >>> dotdot_resolves_lexically('/no/such/dir/../file') # nothing to follow: lexical True
- Return type:
- dol.filesys.ensure_dir(dirpath, *, max_dirs_to_make=None, verbose=False)[source]¶
Ensure that a directory exists, creating it if necessary.
- Parameters:
- Returns:
the path to the directory
When the path does not exist, if
verboseis:a
bool’ a standard message will be printeda
callable; will be called on dirpath before directory is created – you can use this to ask the user for confirmation for examplea ‘’string``; this string will be printed
Usage note: If you want to string or the (argument-less) callable to be dependent on
dirpath, you need make them so when calling ensure_dir.
- dol.filesys.ensure_slash_suffix(path)[source]¶
Add a file separation (/ or ) at the end of path str, if not already present.
An empty path stays empty: an empty prefix has no slash to “ensure”, and turning it into a bare separator anchors otherwise-absolute keys to the filesystem root. On Windows that produces invalid paths like
\C:\Users\...(a separator before the drive letter ->OSError: [Errno 22]); e.g.Files("")used with absolute keys, as indol.misc.get_obj.
- dol.filesys.iter_dirpaths_in_folder_recursively(root_folder, max_levels=None, _current_level=0, include_hidden=False)[source]¶
Recursively generates dirpaths of folder (and subfolders, etc.) up to a given level
- dol.filesys.iter_filepaths_in_folder_recursively(root_folder, max_levels=None, _current_level=0, include_hidden=False)[source]¶
Recursively generates filepaths of folder (and subfolders, etc.) up to a given level
- dol.filesys.mk_absolute_path(path_format)[source]¶
Expand a leading
~, or make a leading.path absolute; other paths are returned as is.
- dol.filesys.mk_dirs_if_missing(store_cls=None, *, max_dirs_to_make=None, verbose=False, key_condition=None, __module__=None, __name__=None, __qualname__=None, __doc__=None, __annotations__=None, __defaults__=None, __kwdefaults__=None)[source]¶
Store decorator that will make the store create directories on write as needed.
Note that it’ll only effect paths relative to the rootdir, which needs to be ensured to exist separatedly.
The directories are made just before the write. If the write then fails (a layer below refuses the key, e.g. a
filt_iterscope, or the write itself errors), the directories this call made are removed again (those still empty), so a rejected key leaves no stray folder behind (i2mint/dol#122). (A concurrent writer that had just seen such a directory exist can still lose it to this cleanup: its write then fails like any write to a missing directory.)
- dol.filesys.mk_dirs_if_missing_preset(self, k, v, *, max_dirs_to_make=None, verbose=False)[source]¶
Preset function that will make the store create directories on write as needed.
It only makes the directory: the write itself is left to the wrapper that calls this preset, so it happens exactly once and its errors propagate (#112). The directory is made only when the innermost layer is a file-system store, the key resolves (through every layer’s
_id_of_key) to a path whose directory is missing, and that layer’svalidate_keyaccepts it – so an invalid key (e.g. one leavingrootdir, #107) never gets a directory, and the write raises for it.(
mk_dirs_if_missingitself no longer goes through this preset: it also has to see the write fail, to remove the directories it made for a key a layer below rejects, i2mint/dol#122.)
- dol.filesys.mk_json_bytes_wrap(*, loads_kwargs=None, dumps_kwargs=None)[source]¶
Make a
wrap_kvsvalue-codec wrapper for JSON, with kwargs forjson.loads/json.dumps.- Return type:
- dol.filesys.mk_pickle_bytes_wrap(*, loads_kwargs=None, dumps_kwargs=None)[source]¶
Make a
wrap_kvsvalue-codec wrapper for pickle, with kwargs forpickle.loads/pickle.dumps.- Return type:
- dol.filesys.mk_tmp_dol_dir(dirname='', make_it_if_necessary=True, verbose=False)¶
Create and return a path to a temporary directory that’s guaranteed to be accessible to the user.
- Parameters:
dirname – Optional subdirectory name to append to the temporary directory path
make_it_if_necessary – Whether to create the directory if it doesn’t exist
verbose – Controls verbosity when creating directories
- Returns:
Path to a temporary directory that the user has access to
Note
This function creates a user-specific temporary directory to avoid permission issues with system-wide temporary directories.
- dol.filesys.path_is_under(path, rootdir)[source]¶
Whether
path, lexically normalized, isrootdiror lies under it.A
startswithcheck is not enough:'/data/root/../secret'starts with'/data/root/'but addresses/data/secret(i2mint/dol#107). The check is lexical (os.path.normpath, notrealpath), so a symlinked sub-directory of the root still counts as inside it.>>> path_is_under('/data/root/a/b.txt', '/data/root/') True >>> path_is_under('/data/root/a/../b.txt', '/data/root/') True >>> path_is_under('/data/root/../secret.txt', '/data/root/') False >>> path_is_under('/data/root/a/../../secret.txt', '/data/root') False >>> path_is_under('/data/rootsibling/x', '/data/root') False >>> path_is_under('//data/root/a', '//data/root/') True >>> path_is_under('root/../a', 'root/..'), path_is_under('root/../../a', 'root/..') (True, False)
- Return type:
- dol.filesys.path_of(store, key)[source]¶
The file-system path of
keyin a file-backedstore, however it is wrapped.The key is followed through every layer’s
_id_of_key(the same walk asdol.inner_most_key) down to the innermost file-system store, which must accept the result: a key leavingrootdirraisesKeyValidationError(aKeyError), as a read or write of it would. A store with no file-system layer at its core, or with a layer that maps keys without defining_id_of_key(so the key cannot be followed), raisesTypeError. Nothing is read or created. The result is an absolute, normalized path (relative roots resolve against the CWD).Limit: like
dol.dig.key_in_scope, this reads what the layers declare through_id_of_key; a layer that maps keys by other means (add_path_access,flatten) is not followed, and afilt_iterscope is not enforced.>>> import tempfile >>> from dol import Files, KeyCodecs >>> root = tempfile.mkdtemp() >>> path_of(Files(root), os.path.join('a', 'b.txt')) == os.path.join(root, 'a', 'b.txt') True >>> s = KeyCodecs.suffixed('.json')(Files(root)) # wrapped: the outer key is mapped >>> path_of(s, 'a').endswith('a.json') True >>> path_of(Files(root), '../x') Traceback (most recent call last): ... dol.filesys.KeyValidationError: 'Key not valid (usually because does not exist or access not permitted): ...' >>> path_of({}, 'a') Traceback (most recent call last): ... TypeError: ...
- Return type:
- dol.filesys.paths_in_dir(rootdir, include_hidden=False)[source]¶
Yield the paths of the entries of
rootdir(directories with a trailing separator), skipping hidden ones unlessinclude_hidden.
- dol.filesys.process_path(*path, ensure_dir_exists=False, assert_exists=False, ensure_endswith_slash=False, ensure_does_not_end_with_slash=False, expanduser=True, expandvars=True, abspath=True, rootdir='')[source]¶
Process a path string, ensuring it exists, and optionally expanding user.
- Parameters:
path (
Iterable[str]) – The path to process. Can be multiple components of a path.ensure_dir_exists (
int|bool) – Whether to ensure the path exists.assert_exists (
bool) – Whether to assert that the path exists.ensure_endswith_slash (
bool) – Whether to ensure the path ends with a slash.ensure_does_not_end_with_slash (
bool) – Whether to ensure the path does not end with a slash.expanduser (
bool) – Whether to expand the user in the path.expandvars (
bool) – Whether to expand environment variables in the path.abspath (
bool) – Whether to convert the path to an absolute path.rootdir (
str) – The root directory to prepend to the path.
- Returns:
The processed path.
- Return type:
The result uses the running OS’s native separator, so these examples assert OS-independently (the literal forward-slash form is what you get on POSIX):
>>> import os >>> process_path('a', 'b', 'c').endswith(os.path.join('a', 'b', 'c')) True >>> p = process_path( ... 'a', 'b', 'c', rootdir='root_dir', ... ensure_endswith_slash=True, abspath=False, expanduser=False, expandvars=False, ... ) >>> p == os.path.join('root_dir', 'a', 'b', 'c') + os.sep True
- dol.filesys.resolve_dir(dirpath, assert_existence=False, ensure_existence=False)[source]¶
Resolve a path to a full, real, path to a directory
- dol.filesys.resolve_path(path, assert_existence=False)[source]¶
Resolve a path to a full, real, (file or folder) path (opt assert existence). That is, resolve situations where ~ and . prefix the paths.
- dol.filesys.subfolder_stores(root_folder, *, max_levels=None, include_hidden=False, relative_paths=True, slash_suffix=False, folder_to_store=<class 'dol.filesys.Files'>)[source]¶
Create a store of subfolders of a given folder, where the keys are the subfolder paths (by default, relative and slash-less) and the values are stores of these subfolders.
By default, all subfolders will be taken, recursively, but this can be controlled by the
max_levelsparameter.
- dol.filesys.temp_dir(dirname='', make_it_if_necessary=True, verbose=False)[source]¶
Create and return a path to a temporary directory that’s guaranteed to be accessible to the user.
- Parameters:
dirname – Optional subdirectory name to append to the temporary directory path
make_it_if_necessary – Whether to create the directory if it doesn’t exist
verbose – Controls verbosity when creating directories
- Returns:
Path to a temporary directory that the user has access to
Note
This function creates a user-specific temporary directory to avoid permission issues with system-wide temporary directories.
- dol.filesys.validate_key_and_raise_key_error_on_exception(func)[source]¶
Method decorator: validate the key first, and re-raise any exception of the method as a
KeyError.Which exceptions become a
KeyErroris the store’s_reads_as_missing(error, k)(default: everyException; seeFileBytesReader(strict_key_errors=...)).