dol.paths

Module for path (and path-like) object manipulation

Examples:

>>> d = {'a': {'b': {'c': 1, 'd': 2}, 'e': 3}}
>>> list(path_filter(lambda p, k, v: v == 2, d))
[('a', 'b', 'd')]
>>> path_get(d, ('a', 'b', 'd'))
2
>>> path_set(d, ('a', 'b', 'd'), 4)
>>> d
{'a': {'b': {'c': 1, 'd': 4}, 'e': 3}}
>>> path_set(d, ('a', 'b', 'new_ab_key'), 42)
>>> d
{'a': {'b': {'c': 1, 'd': 4, 'new_ab_key': 42}, 'e': 3}}

class dol.paths.Codec(encoder, decoder)

decoder

Alias for field number 1

encoder

Alias for field number 0

class dol.paths.KeyPath(path_sep: str = '/', _path_type: Union[type, callable] = <class 'tuple'>)

A key mapper that converts from an iterable key (default tuple) to a string (given a path-separator str)

Parameters

• path_sep – The path separator (used to make string paths from iterable paths and visa versa

• _path_type – The type of the outcoming (inner) path. But really, any function to

• from a list to (convert) – the outer path type we want.

With '/' as a separator:

>>> kp = KeyPath(path_sep='/')
>>> kp._key_of_id(('a', 'b', 'c'))
'a/b/c'
>>> kp._id_of_key('a/b/c')
('a', 'b', 'c')

With '.' as a separator:

>>> kp = KeyPath(path_sep='.')
>>> kp._key_of_id(('a', 'b', 'c'))
'a.b.c'
>>> kp._id_of_key('a.b.c')
('a', 'b', 'c')
>>> kp = KeyPath(path_sep=':::', _path_type=dict.fromkeys)
>>> _id = dict.fromkeys('abc')
>>> _id
{'a': None, 'b': None, 'c': None}
>>> kp._key_of_id(_id)
'a:::b:::c'
>>> kp._id_of_key('a:::b:::c')
{'a': None, 'b': None, 'c': None}

Calling a KeyPath instance on a store wraps it so we can have path access to it.

>>> s = {'a': {'b': {'c': 42}}}
>>> s['a']['b']['c']
42
>>> # Now let's wrap the store
>>> s = KeyPath('.')(s)
>>> s['a.b.c']
42
>>> s['a.b.c'] = 3.14
>>> s['a.b.c']
3.14
>>> del s['a.b.c']
>>> s
{'a': {'b': {}}}

Note: KeyPath enables you to read with paths when all the keys of the paths are valid (i.e. have a value), but just as with a dict, it will not create intermediate nested values for you (as for example, you could make for yourself using collections.defaultdict).

class dol.paths.PathKeyTypes(value)

An enumeration.

class dol.paths.PrefixRelativization(_prefix='')

A key wrap that allows one to interface with absolute paths through relative paths. The original intent was for local files. Instead of referencing files through an absolute path such as:

/A/VERY/LONG/ROOT/FOLDER/the/file/we.want

we can instead reference the file as:

the/file/we.want

But PrefixRelativization can be used, not only for local paths, but when ever a string reference is involved. In fact, not only strings, but any key object that has a __len__, __add__, and subscripting.

class dol.paths.PrefixRelativizationMixin

Mixin that adds a intercepts the _id_of_key an _key_of_id methods, transforming absolute keys to relative ones. Designed to work with string keys, where absolute and relative are relative to a _prefix attribute (assumed to exist). The cannonical use case is when keys are absolute file paths, but we want to identify data through relative paths. Instead of referencing files through an absolute path such as

/A/VERY/LONG/ROOT/FOLDER/the/file/we.want

we can instead reference the file as

the/file/we.want

Note though, that PrefixRelativizationMixin can be used, not only for local paths, but when ever a string reference is involved. In fact, not only strings, but any key object that has a __len__, __add__, and subscripting.

When subclassed, should be placed before the class defining _id_of_key an _key_of_id. Also, assumes that a (string) _prefix attribute will be available.

>>> from dol.base import Store
>>> from collections import UserDict
>>>
>>> class MyStore(PrefixRelativizationMixin, Store):
...     def __init__(self, store, _prefix='/root/of/data/'):
...         super().__init__(store)
...         self._prefix = _prefix
...
>>> s = MyStore(store=dict())  # using a dict as our store
>>> s['foo'] = 'bar'
>>> assert s['foo'] == 'bar'
>>> s['too'] = 'much'
>>> assert list(s.keys()) == ['foo', 'too']
>>> # Everything looks normal, but are the actual keys behind the hood?
>>> s._id_of_key('foo')
'/root/of/data/foo'
>>> # see when iterating over s.items(), we get the interface view:
>>> list(s.items())
[('foo', 'bar'), ('too', 'much')]
>>> # but if we ask the store we're actually delegating the storing to, we see what the keys actually are.
>>> s.store.items()
dict_items([('/root/of/data/foo', 'bar'), ('/root/of/data/too', 'much')])

class dol.paths.StringTemplate(template: str, *, field_patterns: dict = None, simple_str_sep: str = None, namedtuple_type_name: str = 'NamedTuple')

A class for parsing and generating strings based on a template.

Parameters

• template – A template string with fields to be extracted or filled in.

• field_patterns – A dictionary of field names and their regex patterns.

• simple_str_sep – A separator string for simple strings (i.e. strings without fields). If None, the template string will be used as the separator.

Examples

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.str_to_dict("Alice is 30 years old.")
{'name': 'Alice', 'age': '30'}
>>> st.dict_to_str({'name': 'Alice', 'age': '30'})
'Alice is 30 years old.'
>>> st.dict_to_tuple({'name': 'Alice', 'age': '30'})
('Alice', '30')
>>> st.tuple_to_dict(('Alice', '30'))
{'name': 'Alice', 'age': '30'}
>>> st.str_to_tuple("Alice is 30 years old.")
('Alice', '30')

You can also ask any (handled) combination of field types: >>> coder, encoder = st.codec(‘tuple’, ‘dict’) >>> coder((‘Alice’, ‘30’)) {‘name’: ‘Alice’, ‘age’: ‘30’} >>> encoder({‘name’: ‘Alice’, ‘age’: ‘30’}) (‘Alice’, ‘30’)

codec(source: Literal[str, dict, tuple, namedtuple, simple_str], target: Literal[str, dict, tuple, namedtuple, simple_str])

Makes a (coder, decoder) pair for the given source and target types.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> coder, encoder = st.codec('tuple', 'dict')
>>> coder(('Alice', '30'))
{'name': 'Alice', 'age': '30'}
>>> encoder({'name': 'Alice', 'age': '30'})
('Alice', '30')

dict_to_namedtuple(params: dict)

Generates a namedtuple from the dictionary values based on the template.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> Person = st.dict_to_namedtuple({'name': 'Alice', 'age': '30'})
>>> Person
NamedTuple(name='Alice', age='30')

dict_to_str(params: dict) → str

Generates a string from the dictionary values based on the template.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.dict_to_str({'name': 'Alice', 'age': '30'})
'Alice is 30 years old.'

dict_to_tuple(params: dict) → tuple

Generates a tuple from the dictionary values based on the template.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.dict_to_tuple({'name': 'Alice', 'age': '30'})
('Alice', '30')

namedtuple_to_dict(nt)

Converts a namedtuple to a dictionary.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> Person = st.dict_to_namedtuple({'name': 'Alice', 'age': '30'})
>>> st.namedtuple_to_dict(Person)
{'name': 'Alice', 'age': '30'}

simple_str_to_str(ss: str, sep: str)

Converts a simple character-delimited string to a string.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.simple_str_to_str('Alice-30', '-')
'Alice is 30 years old.'

str_to_dict(s: str) → dict

Parses the input string and returns a dictionary of extracted values.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )

>>> st.str_to_dict("Alice is 30 years old.")
{'name': 'Alice', 'age': '30'}

str_to_simple_str(s: str, sep: str)

Converts a string to a simple string (i.e. a simple character-delimited string).

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.str_to_simple_str("Alice is 30 years old.", '-')
'Alice-30'

str_to_tuple(s: str) → tuple

Parses the input string and returns a tuple of extracted values.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.str_to_tuple("Alice is 30 years old.")
('Alice', '30')

tuple_to_dict(param_vals: tuple) → dict

Generates a dictionary from the tuple values based on the template.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.tuple_to_dict(('Alice', '30'))
{'name': 'Alice', 'age': '30'}

tuple_to_str(params: tuple) → str

Generates a string from the tuple values based on the template.

>>> st = StringTemplate(
...     "{name} is {age} years old.",
...     field_patterns={"name": r"\w+", "age": r"\d+"}
... )
>>> st.tuple_to_str(('Alice', '30'))
'Alice is 30 years old.'

dol.paths.get_attr_or_item(obj, k)

If k is a string, tries to get k as an attribute of obj first, and if that fails, gets it as obj[k]

dol.paths.mk_relative_path_store(store_cls=None, *, name=None, with_key_validation=False, prefix_attr='_prefix', __module__=None, __name__=None, __qualname__=None, __doc__=None, __annotations__=None, __defaults__=None, __kwdefaults__=None)

Parameters

• store_cls – The base store to wrap (subclass)

• name – The name of the new store (by default ‘RelPath’ + store_cls.__name__)

• with_key_validation – Whether keys should be validated upon access (store_cls must have an is_valid_key method

Returns: A new class that uses relative paths (i.e. where _prefix is automatically added to incoming keys,

and the len(_prefix) first characters are removed from outgoing keys.

>>> # The dynamic way (if you try this at home, be aware of the pitfalls of the dynamic way
>>> # -- but don't just believe the static dogmas).
>>> MyStore = mk_relative_path_store(dict)  # wrap our favorite store: A dict.
>>> s = MyStore()  # make such a store
>>> s._prefix = '/ROOT/'
>>> s['foo'] = 'bar'
>>> dict(s.items())  # gives us what you would expect
{'foo': 'bar'}
>>>  # but under the hood, the dict we wrapped actually contains the '/ROOT/' prefix
>>> dict(s.store)
{'/ROOT/foo': 'bar'}
>>>
>>> # The static way: Make a class that will integrate the _prefix at construction time.
>>> class MyStore(mk_relative_path_store(dict)):  # Indeed, mk_relative_path_store(dict) is a class you can subclass
...     def __init__(self, _prefix, *args, **kwargs):
...         self._prefix = _prefix

You can choose the name you want that prefix to have as an attribute (we’ll still make a hidden ‘_prefix’ attribute for internal use, but at least you can have an attribute with the name you want.

>>> MyRelStore = mk_relative_path_store(dict, prefix_attr='rootdir')
>>> s = MyRelStore()
>>> s.rootdir = '/ROOT/'

>>> s['foo'] = 'bar'
>>> dict(s.items())  # gives us what you would expect
{'foo': 'bar'}
>>>  # but under the hood, the dict we wrapped actually contains the '/ROOT/' prefix
>>> dict(s.store)
{'/ROOT/foo': 'bar'}

dol.paths.path_edit(d: Mapping, edits: Union[Mapping[Union[Iterable, str], VT], Iterable[Tuple[Union[Iterable, str], VT]]] = ()) → Mapping

Make a series of (in place) edits to a Mapping, specifying (path, value) pairs.

Parameters

• d (Mapping) – The mapping to edit.

• edits – An iterable of (path, value) tuples, or path: value Mapping.

Returns

The edited mapping.

Return type

Mapping

>>> d = {'a': 1}
>>> path_edit(d, [(['b', 'c'], 2), ('d.e.f', 3)])
{'a': 1, 'b': {'c': 2}, 'd': {'e': {'f': 3}}}

Changes happened also inplace (so if you don’t want that, make a deepcopy first):

>>> d
{'a': 1, 'b': {'c': 2}, 'd': {'e': {'f': 3}}}

You can also pass a dict of edits.

>>> path_edit(d, {'a': 4, 'd.e.f': 5})
{'a': 4, 'b': {'c': 2}, 'd': {'e': {'f': 5}}}

dol.paths.path_filter(pkv_filt: Callable[[PT, KT, VT], bool], d: Mapping) → Iterator[PT]

Walk a dict, yielding paths to values that pass the pkv_filt

Parameters

• pkv_filt – A function that takes a path, key, and value, and returns True if the path should be yielded, and False otherwise

• d – The Mapping to walk (scan through)

Returns

An iterator of paths to values that pass the pkv_filt

Example:

>>> d = {'a': {'b': {'c': 1, 'd': 2}, 'e': 3}}
>>> list(path_filter(lambda p, k, v: v == 2, d))

[(‘a’, ‘b’, ‘d’)]

>>> mm = {
...     'a': {'b': {'c': 42}},
...     'aa': {'bb': {'cc': 'meaning of life'}},
...     'aaa': {'bbb': 314},
... }
>>> return_path_if_int_leaf = lambda p, k, v: (p, v) if isinstance(v, int) else None
>>> paths = list(path_filter(return_path_if_int_leaf, mm))
>>> paths  # only the paths to the int leaves are returned
[('a', 'b', 'c'), ('aaa', 'bbb')]

The pkv_filt argument can use path, key, and/or value to define your search query. For example, let’s extract all the paths that have depth at least 3.

>>> paths = list(path_filter(lambda p, k, v: len(p) >= 3, mm))
>>> paths
[('a', 'b', 'c'), ('aa', 'bb', 'cc')]

The rationale for path_filter yielding matching paths, and not values or keys, is that if you have the paths, you can than get the keys and values with them, using path_get.

>>> from functools import partial, reduce
>>> path_get = lambda m, k: reduce(lambda m, k: m[k], k, m)
>>> extract_paths = lambda m, paths: map(partial(path_get, m), paths)
>>> vals = list(extract_paths(mm, paths))
>>> vals
[42, 'meaning of life']

dol.paths.path_get(obj: Any, path, on_error: Union[Callable[[dict], Any], str] = <function raise_on_error>, *, sep='.', key_transformer=<function cast_to_int_if_numeric_str>, get_value: Callable = <function get_attr_or_item>, caught_errors=(<class 'Exception'>,))

Get elements of a mapping through a path to be called recursively.

It will

• split a path into keys if it is a string, using the specified seperator sep

• consider string keys that are numeric as ints (convenient for lists)

• get items also as attributes (attributes are checked for first for string keys)

• catch all exceptions (that are subclasses of Exception)

>>> class A:
...      an_attribute = 42
>>> path_get([1, [4, 5, {'a': A}], 3], [1, 2, 'a', 'an_attribute'])
42

By default, if path is a string, it will be split on sep, which is '.' by default.

>>> path_get([1, [4, 5, {'a': A}], 3], '1.2.a.an_attribute')
42

Note: The underlying function is _path_get, but path_get has defaults and flexible input processing for more convenience.

Note: path_get contains some ready-made OnErrorType functions in its attributes. For example, see how we can make path_get have the same behavior as dict.get by passing path_get.return_none_on_error as on_error:

>>> dd = path_get({}, 'no.keys', on_error=path_get.return_none_on_error)
>>> dd is None
True

For example, path_get.raise_on_error, path_get.return_none_on_error, and path_get.return_empty_tuple_on_error.

dol.paths.path_set(d: Mapping, key_path: Iterable, val: VT, *, sep: str = '.', new_mapping: Callable[[], VT] = <class 'dict'>)

Sets a val to a path of keys.

Parameters

• d – The mapping to set the value in

• key_path – The path of keys to set the value to

• val – The value to set

• sep – The separator to use if the path is a string

• new_mapping – callable that returns a new mapping to use when key is not found

Returns

>>> d = {'a': 1, 'b': {'c': 2}}
>>> path_set(d, ['b', 'e'], 42)
>>> d
{'a': 1, 'b': {'c': 2, 'e': 42}}

>>> input_dict = {
...   "a": {
...     "c": "val of a.c",
...     "b": 1,
...   },
...   "10": 10,
...   "b": {
...     "B": {
...       "AA": 3
...     }
...   }
... }
>>>
>>> path_set(input_dict, ('new', 'key', 'path'), 7)
>>> input_dict  
{'a': {'c': 'val of a.c', 'b': 1}, '10': 10, 'b': {'B': {'AA': 3}},
'new': {'key': {'path': 7}}}

You can also use a string as a path, with a separator:

>>> path_set(input_dict, 'new/key/old/path', 8, sep='/')
>>> input_dict  
{'a': {'c': 'val of a.c', 'b': 1}, '10': 10, 'b': {'B': {'AA': 3}},
'new': {'key': {'path': 7, 'old': {'path': 8}}}}

If you specify a string path and a non-None separator, the separator will be used to split the string into a list of keys. The default separator is sep='.'.

>>> path_set(input_dict, 'new.key', 'new val')
>>> input_dict  
{'a': {'c': 'val of a.c', 'b': 1}, '10': 10, 'b': {'B': {'AA': 3}},
'new': {'key': 'new val'}}

You can also specify a different new_mapping factory, which will be used to create new mappings when a key is missing. The default is dict.

>>> from collections import OrderedDict
>>> input_dict = {}
>>> path_set(input_dict, 'new.key', 42, new_mapping=OrderedDict)
>>> input_dict  
{'new': OrderedDict([('key', 42)])}

dol.paths.rel_path_wrap(o, _prefix)

Parameters

• o – An object to be wrapped

• _prefix – The _prefix to use for key wrapping (will remove it from outcoming keys and add to ingoing keys.

>>> # The dynamic way (if you try this at home, be aware of the pitfalls of the dynamic way
>>> # -- but don't just believe the static dogmas).
>>> d = {'/ROOT/of/every/thing': 42, '/ROOT/of/this/too': 0}
>>> dd = rel_path_wrap(d, '/ROOT/of/')
>>> dd['foo'] = 'bar'
>>> dict(dd.items())  # gives us what you would expect
{'every/thing': 42, 'this/too': 0, 'foo': 'bar'}
>>>  # but under the hood, the dict we wrapped actually contains the '/ROOT/' prefix
>>> dict(dd.store)
{'/ROOT/of/every/thing': 42, '/ROOT/of/this/too': 0, '/ROOT/of/foo': 'bar'}
>>>
>>> # The static way: Make a class that will integrate the _prefix at construction time.
>>> class MyStore(mk_relative_path_store(dict)):  # Indeed, mk_relative_path_store(dict) is a class you can subclass
...     def __init__(self, _prefix, *args, **kwargs):
...         self._prefix = _prefix

dol.paths.search_paths(d: Mapping, pkv_filt: Callable[[PT, KT, VT], bool]) → Iterator[PT]

backwards compatibility quasi-alias (arguments are flipped) Use path_filter instead, since search_paths will be deprecated.

dol.paths.str_template_key_trans(template: str, key_type: Union[dol.paths.PathKeyTypes, type], format_dict=None, process_kwargs=None, process_info_dict=None, named_tuple_type_name='NamedTuple', sep: str = '/')

Make a key trans object that translates from a string _id to a dict, tuple, or namedtuple key (and back)