> built 2026-09-22 13:56 UTC from 66e3056 (master) · dol 0.3.70. Details: build_info.json # index.html.md # dol Base builtin tools make and transform data object layers (dols). The main idea comes in many names such as [Data Access Object (DAO)](https://en.wikipedia.org/wiki/Data_access_object), [Repository Pattern](https://www.cosmicpython.com/book/chapter_02_repository.html), [Hexagonal architecture, or ports and adapters architecture](https://en.wikipedia.org/wiki/Hexagonal_architecture_(software)) for data. But simply put, what `dol` provides is tools to make your interface with data be domain-oriented, simple, and isolated from the underlying data infrastucture. This makes the business logic code simple and stable, enables you to develop and test it without the need of any data infrastructure, and allows you to change this infrastructure independently. The package is light-weight: Pure python; no third-party dependencies. To install: `pip install dol` [Documentation here](https://i2mint.github.io/dol/) ## For AI agents `dol` ships tooling for coding agents. If you are one, start here. **Skills** ([Agent Skills](https://agentskills.io) format), for any agent host. | Skill | Use it to | |-----------------------|------------------------------------------------------------------------------------------------------------------------------------| | `dol-dev-portability` | keep dol working on Windows as well as Linux/macOS | | `dol-dev-wrap-kvs` | understand and safely modify dol’s core wrapping machinery — wrap_kvs, store_decorator, Store.wrap, and how transforms are applied | | `dol-store-building` | build a dol store: wrap any storage backend | **Instruction files**: `CLAUDE.md` (Claude Code). **The documentation, machine-readable**: [`llms.txt`](https://i2mint.github.io/dol/llms.txt) indexes every page; [`dol.md`](https://i2mint.github.io/dol/dol.md) is the whole documentation in one file; every page has a `.md` twin; [`objects.inv`](https://i2mint.github.io/dol/objects.inv) maps symbols to URLs. The full list, with install lines, is on the site’s [For AI agents](https://i2mint.github.io/dol/ai-agents.html) page. If you like writing your own code, the rest of this README is written for you, starting at [Example use](). ## Example use Say you have a source backend that has pickles of some lists-of-lists-of-strings, using the `.pkl` extension, and you want to copy this data to a target backend, but saving them as gzipped csvs with the `csv.gz` extension. We’ll first work with dictionaries instead of files here, so we can test more easily, and safely. ```python import pickle src_backend = { "file_1.pkl": pickle.dumps([["A", "B", "C"], ["one", "two", "three"]]), "file_2.pkl": pickle.dumps([["apple", "pie"], ["one", "two"], ["hot", "cold"]]), } targ_backend = dict() ``` Here’s how you can do it with `dol` tools ```python from dol import ValueCodecs, KeyCodecs, Pipe # decoder here will unpickle data and remove remove the .pkl extension from the key src_wrap = Pipe(KeyCodecs.suffixed(".pkl"), ValueCodecs.pickle()) # encoder here will convert the lists to csv string, the string into bytes, # and the bytes will be gzipped. # ... also, we'll add .csv.gz on write. targ_wrap = Pipe( KeyCodecs.suffixed(".csv.gz"), ValueCodecs.csv() + ValueCodecs.str_to_bytes() + ValueCodecs.gzip(), ) # Let's wrap our backends: src = src_wrap(src_backend) targ = targ_wrap(targ_backend) # and copy src over to targ print(f"Before: {list(targ_backend)=}") targ.update(src) print(f"After: {list(targ_backend)=}") ``` From the point of view of src and targ, you see the same thing. ```python assert list(src) == list(targ) == ["file_1", "file_2"] assert src["file_1"] == targ["file_1"] == [["A", "B", "C"], ["one", "two", "three"]] ``` But the backend of targ is different: ```python src_backend["file_1.pkl"] # b'\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00]\x94(]\x94(K\x01K\x02K\x03e]\x94(K\x04K\x05K\x06ee.' targ_backend["file_1.csv.gz"] # b'\x1f\x8b\x08\x00*YWe\x02\xff3\xd41\xd21\xe6\xe52\xd11\xd51\xe3\xe5\x02\x00)4\x83\x83\x0e\x00\x00\x00' ``` Now that you’ve tested your setup with dictionaries, you’re ready to move on to real, persisted storage. If you wanted to do this with local files, you’d: ```python from dol import Files src = Files('PATH_TO_LOCAL_SOURCE_FOLDER') targ = Files('PATH_TO_LOCAL_TARGET_FOLDER) ``` But you could do this with AWS S3 using tools from [s3dol](https://github.com/i2mint/s3dol), or Azure using tools from [azuredol](https://github.com/i2mint/azuredol), or mongoDB with [mongodol](https://github.com/i2mint/mongodol), github with [hubcap](https://github.com/thorwhalen/hubcap), and so on… All of these extensions provide adapters from various data sources/targets to the dict-like interface (called “Mapping” in python typing). What `dol` provides are base tools to make a path from these to the interface that makes sense for the domain, or business logic in front of you, so that you can purify your code from implementation details, and therefore be create more robust and flexible code as far as data operations are concerned. ## A list various packages that use dol `py2store` provides tools to create the dict-like interface to data you need. If you want to just use existing interfaces, build on it, or find examples of how to make such interfaces, check out the ever-growing list of `py2store`-using projects: - [mongodol](https://github.com/i2mint/mongodol): For MongoDB - [tabled](https://github.com/i2mint/tabled): Data as `pandas.DataFrame` from various sources - [msword](https://pypi.org/project/msword/): Simple mapping view to docx (Word Doc) elements - [sshdol](https://github.com/i2mint/sshdol): Remote (ssh) files access - [haggle](https://github.com/otosense/haggle): Easily search, download, and use kaggle datasets. - [pyckup](https://github.com/i2mint/pyckup): Grab data simply and define protocols for others to do the same. - [hubcap](https://pypi.org/project/hubcap/): Dict-like interface to github. - [graze](https://github.com/thorwhalen/graze): Cache the internet. - [grub](https://github.com/thorwhalen/grub): A ridiculously simple search engine maker. - [hear](https://github.com/otosense/hear): Read/write audio data flexibly. Just for fun projects: - [cult](https://github.com/thorwhalen/cult): Religious texts search engine. 18mn application of `grub`. - [laugh](https://github.com/thorwhalen/laugh): A (py2store-based) joke finder. # Caching # Use cases ## Interfacing reads How many times did someone share some data with you in the form of a zip of some nested folders whose structure and naming choices are fascinatingly obscure? And how much time do you then spend to write code to interface with that freak of nature? Well, one of the intents of py2store is to make that easier to do. You still need to understand the structure of the data store and how to deserialize these datas into python objects you can manipulate. But with the proper tool, you shouldn’t have to do much more than that. ## Changing where and how things are stored Ever have to switch where you persist things (say from file system to S3), or change the way key into your data, or the way that data is serialized? If you use py2store tools to separate the different storage concerns, it’ll be quite easy to change, since change will be localized. And if you’re dealing with code that was already written, with concerns all mixed up, py2store should still be able to help since you’ll be able to more easily give the new system a facade that makes it look like the old one. All of this can also be applied to data bases as well, in-so-far as the CRUD operations you’re using are covered by the base methods. ## Adapters: When the learning curve is in the way of learning Shinny new storage mechanisms (DBs etc.) are born constantly, and some folks start using them, and we are eventually lead to use them as well if we need to work with those folks’ systems. And though we’d love to learn the wonderful new capabilities the new kid on the block has, sometimes we just don’t have time for that. Wouldn’t it be nice if someone wrote an adapter to the new system that had an interface we were familiar with? Talking to SQL as if it were mongo (or visa versa). Talking to S3 as if it were a file system. Now it’s not a long term solution: If we’re really going to be using the new system intensively, we should learn it. But when you just got to get stuff done, having a familiar facade to something new is a life saver. py2store would like to make it easier for you roll out an adapter to be able to talk to the new system in the way **you** are familiar with. ## Thinking about storage later, if ever You have a new project or need to write a new app. You’ll need to store stuff and read stuff back. Stuff: Different kinds of resources that your app will need to function. Some people enjoy thinking of how to optimize that aspect. I don’t. I’ll leave it to the experts to do so when the time comes. Often though, the time is later, if ever. Few proof of concepts and MVPs ever make it to prod. So instead, I’d like to just get on with the business logic and write my program. So what I need is an easy way to get some minimal storage functionality. But when the time comes to optimize, I shouldn’t have to change my code, but instead just change the way my DAO does things. What I need is py2store. ## Remove data access entropy Data comes from many different sources, organization, and formats. Data is needed in many different contexts, which comes with its own natural data organization and formats. In between both: A entropic mess of ad-hoc connections and annoying time-consuming and error prone boilerplate. `py2store` (and it’s now many extensions) is there to mitigate this. The design gods say SOC, DRY, SOLID\* and such. That’s good design, yes. But it can take more work to achieve these principles. We’d like to make it *easier* to do it right than do it wrong. *(\*) Separation (Of) Concerns, Don’t Repeat Yourself, https://en.wikipedia.org/wiki/SOLID))* We need to determine what are the most common operations we want to do on data, and decide on a common way to express these operations, no matter what the implementation details are. - get/read some data - set/write some data - list/see what data we have - filter - cache … Looking at this, we see that the base operations for complex data systems such as data bases and file systems overlap significantly with the base operations on python (or any programming language) objects. So we’ll reflect this in our choice of a common “language” for these operations. For examples, once projected to a `py2store` object, iterating over the contents of a data base, or over files, or over the elements of a python (iterable) object should look the same, in code. Achieving this, we achieve SOC, but also set ourselves up for tooling that can assume this consistency, therefore be DRY, and many of the SOLID principles of design. Also mentionable: So far, `py2store` core tools are all pure python – no dependencies on anything else. Now, when you want to specialize a store (say talk to data bases, web services, acquire special formats (audio, etc.)), then you’ll need to pull in a few helpful packages. But the core tooling is pure. # A few words about design By store we mean key-value store. This could be files in a filesystem, objects in s3, or a database. Where and how the content is stored should be specified, but StoreInterface offers a dict-like interface to this. ```none __getitem__ calls: _id_of_key _obj_of_data __setitem__ calls: _id_of_key _data_of_obj __delitem__ calls: _id_of_key __iter__ calls: _key_of_id ``` ```python >>> from dol import Store ``` A Store can be instantiated with no arguments. By default it will make a dict and wrap that. ```python >>> # Default store: no key or value conversion ################################################ >>> s = Store() >>> s['foo'] = 33 >>> s['bar'] = 65 >>> assert list(s.items()) == [('foo', 33), ('bar', 65)] >>> assert list(s.store.items()) == [('foo', 33), ('bar', 65)] # see that the store contains the same thing ``` Now let’s make stores that have a key and value conversion layer input keys will be upper cased, and output keys lower cased input values (assumed int) will be converted to ascii string, and visa versa ```python >>> >>> def test_store(s): ... s['foo'] = 33 # write 33 to 'foo' ... assert 'foo' in s # __contains__ works ... assert 'no_such_key' not in s # __nin__ works ... s['bar'] = 65 # write 65 to 'bar' ... assert len(s) == 2 # there are indeed two elements ... assert list(s) == ['foo', 'bar'] # these are the keys ... assert list(s.keys()) == ['foo', 'bar'] # the keys() method works! ... assert list(s.values()) == [33, 65] # the values() method works! ... assert list(s.items()) == [('foo', 33), ('bar', 65)] # these are the items ... assert list(s.store.items()) == [('FOO', '!'), ('BAR', 'A')] # but note the internal representation ... assert s.get('foo') == 33 # the get method works ... assert s.get('no_such_key', 'something') == 'something' # return a default value ... del(s['foo']) # you can delete an item given its key ... assert len(s) == 1 # see, only one item left! ... assert list(s.items()) == [('bar', 65)] # here it is >>> ``` We can introduce this conversion layer in several ways. Here are few… ## by subclassing ```python >>> # by subclassing ############################################################################### >>> class MyStore(Store): ... def _id_of_key(self, k): ... return k.upper() ... def _key_of_id(self, _id): ... return _id.lower() ... def _data_of_obj(self, obj): ... return chr(obj) ... def _obj_of_data(self, data): ... return ord(data) >>> s = MyStore(store=dict()) # note that you don't need to specify dict(), since it's the default >>> test_store(s) >>> ``` ## by assigning functions to converters ```python >>> # by assigning functions to converters ########################################################## >>> class MyStore(Store): ... def __init__(self, store, _id_of_key, _key_of_id, _data_of_obj, _obj_of_data): ... super().__init__(store) ... self._id_of_key = _id_of_key ... self._key_of_id = _key_of_id ... self._data_of_obj = _data_of_obj ... self._obj_of_data = _obj_of_data ... >>> s = MyStore(dict(), ... _id_of_key=lambda k: k.upper(), ... _key_of_id=lambda _id: _id.lower(), ... _data_of_obj=lambda obj: chr(obj), ... _obj_of_data=lambda data: ord(data)) >>> test_store(s) >>> ``` ## using a Mixin class ```python >>> # using a Mixin class ############################################################################# >>> class Mixin: ... def _id_of_key(self, k): ... return k.upper() ... def _key_of_id(self, _id): ... return _id.lower() ... def _data_of_obj(self, obj): ... return chr(obj) ... def _obj_of_data(self, data): ... return ord(data) ... >>> class MyStore(Mixin, Store): # note that the Mixin must come before Store in the mro ... pass ... >>> s = MyStore() # no dict()? No, because default anyway >>> test_store(s) ``` ## adding wrapper methods to an already made Store instance ```python >>> # adding wrapper methods to an already made Store instance ######################################### >>> s = Store(dict()) >>> s._id_of_key=lambda k: k.upper() >>> s._key_of_id=lambda _id: _id.lower() >>> s._data_of_obj=lambda obj: chr(obj) >>> s._obj_of_data=lambda data: ord(data) >>> test_store(s) ``` # And more… ## Why the name? - because it’s short - because it’s cute - because it reminds one of “russian dolls” (one way to think of wrappers) - because we can come up with an acronym the contains “Data Object” in it. ## Historical note Note: This project started as [`py2store`](https://github.com/i2mint/py2store). `dol` is the core of py2store has now been factored out and many of the specialized data object layers moved to separate packages. `py2store` is acting more as an aggregator package – a shoping mall where you can quickly access many (but not all) functionalities that use `dol`. It’s advised to use `dol` (and/or its specialized spin-off packages) directly when the core functionality is all you need.

This documentation as a single file: dol.md (Markdown, for agents).

# _autosummary/dol.appendable.html.md # dol.appendable ### dol.appendable(store_cls=None, , item2kv, return_keys=False, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Makes a new class with append (and consequential extend) methods * **Parameters:** * **store_cls** – The store class to subclass * **item2kv** – The function that produces a (key, val) pair from an item * **new_store_name** – The name to give the new class (default will be ‘Appendable’ + store_cls._\_name_\_) * **Returns:** append, and extend. * **Return type:** A subclass of store_cls with two additional methods ```pycon >>> item_to_kv = lambda item: (item['L'], item) # use value of 'L' as the key, and value is the item itself >>> MyStore = appendable(dict, item2kv=item_to_kv) >>> s = MyStore(); s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items()) [('let', {'L': 'let', 'I': 'it', 'G': 'go'})] ``` Use mk_item2kv.from_item_to_key_params_and_val with tuple key params ```pycon >>> item_to_kv = appendable.mk_item2kv_for.item_to_key_params_and_val(lambda x: ((x['L'], x['I']), x['G']), '{}/{}') >>> MyStore = appendable(item2kv=item_to_kv)(dict) # showing the append(...)(store) form >>> s = MyStore(); s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items()) [('let/it', 'go')] ``` Use mk_item2kv.from_item_to_key_params_and_val with dict key params ```pycon >>> item_to_kv = appendable.mk_item2kv_for.item_to_key_params_and_val( ... lambda x: ({'L': x['L'], 'G': x['G']}, x['I']), '{G}_{L}') >>> @appendable(item2kv=item_to_kv) # showing the @ form ... class MyStore(dict): ... pass >>> s = MyStore(); s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items()) [('go_let', 'it')] ``` Use mk_item2kv.fields to get a tuple key from item fields, defining the sub-dict of the remaining fields to be the value. Also showing here how you can decorate the instance itself. ```pycon >>> item_to_kv = appendable.mk_item2kv_for.fields(['G', 'L'], key_as_tuple=True) >>> d = {} >>> s = appendable(d, item2kv=item_to_kv) >>> s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items()) [(('go', 'let'), {'I': 'it'})] ``` You can make the “append” and “extend” methods to return the new generated keys by using the “return_keys” flag. ```pycon >>> d = {} >>> s = appendable(d, item2kv=item_to_kv, return_keys=True) >>> s.append({'L': 'let', 'I': 'it', 'G': 'go'}) ('go', 'let') ``` # _autosummary/dol.base.html.md # dol.base Base classes for making stores. In the language of the collections.abc module, a store is a MutableMapping that is configured to work with a specific representation of keys, serialization of objects (python values), and persistence of the serialized data. That is, stores offer the same interface as a dict, but where the actual implementation of writes, reads, and listing are configurable. Consider the following example. You’re store is meant to store waveforms as wav files on a remote server. Say waveforms are represented in python as a tuple (wf, sr), where wf is a list of numbers and sr is the sample rate, an int). The \_\_setitem_\_ method will specify how to store bytes on a remote server, but you’ll need to specify how to SERIALIZE (wf, sr) to the bytes that constitute that wav file: \_data_of_obj specifies that. You might also want to read those wav files back into a python (wf, sr) tuple. The \_\_getitem_\_ method will get you those bytes from the server, but the store will need to know how to DESERIALIZE those bytes back into a python object: \_obj_of_data specifies that Further, say you’re storing these .wav files in /some/folder/on/the/server/, but you don’t want the store to use these as the keys. For one, it’s annoying to type and harder to read. But more importantly, it’s an irrelevant implementation detail that shouldn’t be exposed. THe \_id_of_key and \_key_of_id pair are what allow you to add this key interface layer. These key converters object serialization methods default to the identity (i.e. they return the input as is). This means that you don’t have to implement these as all, and can choose to implement these concerns within the storage methods themselves. Main entry points: - `KvReader`: base class for read-only stores (a `Mapping` with a `head`) - `KvPersister`: base class for read-write stores (a `MutableMapping`, `clear` disabled) - `Store`: a persister with the key/value transform hooks, wrapping a backend - `kv_walk`: walk a nested mapping, yielding (path, key, value) triples by default ```pycon >>> from dol.base import Store >>> s = Store({}) >>> s['a'] = 1 >>> s['a'], list(s) (1, ['a']) ``` ### Functions | [`asis`](_autosummary/dol.base.html.md#dol.base.asis)(p, k, v) | Return `(p, k, v)` as is (the default `kv_walk` `leaf_yield`). | |-------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`delegate_to`](_autosummary/dol.base.html.md#dol.base.delegate_to)(wrapped[, class_trans, ...]) | Class decorator factory: the decorated wrapper class constructs a `wrapped` instance and delegates to it, through `delegation_attr`, the attributes of `wrapped` (`dir(wrapped)` minus `ignore`, plus `include`) not already defined on the wrapper. | | [`delegator_wrap`](_autosummary/dol.base.html.md#dol.base.delegator_wrap)(delegator, obj[, ...]) | Wrap a `obj` (type or instance) with `delegator`. | | [`has_kv_store_interface`](_autosummary/dol.base.html.md#dol.base.has_kv_store_interface)(o) | Check if object has the KvStore interface (that is, has the kv wrapper methods | | [`kv_walk`](_autosummary/dol.base.html.md#dol.base.kv_walk)(v[, leaf_yield, walk_filt, ...]) | Walks a nested structure of mappings, yielding stuff on the way. | | [`tuple_keypath_and_val`](_autosummary/dol.base.html.md#dol.base.tuple_keypath_and_val)(p, k, v) | Extend the path `p` with the key `k` and return `(new_path, v)` (the default `kv_walk` `pkv_to_pv`). | | [`val_is_mapping`](_autosummary/dol.base.html.md#dol.base.val_is_mapping)(p, k, v) | Whether the walked value `v` is a `Mapping` (a `kv_walk` `walk_filt`). | | `wrapped_delegator_reconstruct`(wrapped_cls, ...) | | | [`wrapped_self`](_autosummary/dol.base.html.md#dol.base.wrapped_self)(obj) | Return the outermost transform-applying store wrapping `obj`, else `obj` itself. | ### Classes | [`AttrNames`](_autosummary/dol.base.html.md#dol.base.AttrNames)() | Name sets of the methods that make up each mapping interface (`Collection`, `Mapping`, `KvReader`, `KvPersister`, ...). | |-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`Collection`](_autosummary/dol.base.html.md#dol.base.Collection)() | The same as collections.abc.Collection, with some modifications: | | [`DelegatedAttribute`](_autosummary/dol.base.html.md#dol.base.DelegatedAttribute)(delegate_name, attr_name) | Descriptor forwarding `attr_name` lookups to the object held in the instance's `delegate_name` attribute. | | [`KeyValidationABC`](_autosummary/dol.base.html.md#dol.base.KeyValidationABC)() | An ABC for an object writer. | | [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister)() | Acts as a MutableMapping abc, but disabling the clear and \_\_reversed_\_ method, and computing \_\_len_\_ by iterating over all keys, and counting them. | | [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader)() | Acts as a Mapping abc, but with default \_\_len_\_ (implemented by counting keys) and head method to get the first (k, v) item of the store | | [`KvStore`](_autosummary/dol.base.html.md#dol.base.KvStore) | | | [`MappingViewMixin`](_autosummary/dol.base.html.md#dol.base.MappingViewMixin)() | Make `keys()`, `values()` and `items()` build their views from the `KeysView`, `ValuesView` and `ItemsView` class attributes, so a subclass can swap in its own view classes. | | [`NoSuchItem`](_autosummary/dol.base.html.md#dol.base.NoSuchItem)() | Sentinel type; `no_such_item` is its instance. | | [`Persister`](_autosummary/dol.base.html.md#dol.base.Persister) | | | [`Reader`](_autosummary/dol.base.html.md#dol.base.Reader) | | | [`Store`](_autosummary/dol.base.html.md#dol.base.Store)([store]) | By store we mean key-value store. | | [`Stream`](_autosummary/dol.base.html.md#dol.base.Stream)(stream) | A layer-able version of the stream interface | | [`stream_util`](_autosummary/dol.base.html.md#dol.base.stream_util)() | Small callbacks for `Stream`: an always-true filter, a no-op, and rewind (`skip_lines` currently only rewinds). | ### *class* dol.base.AttrNames Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Name sets of the methods that make up each mapping interface (`Collection`, `Mapping`, `KvReader`, `KvPersister`, …). ### *class* dol.base.Collection Bases: [`Collection`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection) The same as collections.abc.Collection, with some modifications: - Addition of a `head` ### *class* dol.base.DelegatedAttribute(delegate_name, attr_name) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Descriptor forwarding `attr_name` lookups to the object held in the instance’s `delegate_name` attribute. ### *class* dol.base.KeyValidationABC Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An ABC for an object writer. Single purpose: store an object under a given key. How the object is serialized and or physically stored should be defined in a concrete subclass. ### *class* dol.base.KvPersister Bases: [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader), [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping) Acts as a MutableMapping abc, but disabling the clear and \_\_reversed_\_ method, and computing \_\_len_\_ by iterating over all keys, and counting them. Note that KvPersister is a MutableMapping, and as such, is dict-like. But that doesn’t mean it’s a dict. For instance, consider the following code: ```python s = SomeKvPersister() s['a']['b'] = 3 ``` If `s` is a dict, this would have the effect of adding a (‘b’, 3) item under ‘a’. But in the general case, this might - fail, because the `s['a']` doesn’t support sub-scripting (doesn’t have a `__getitem__`) - or, worse, will pass silently but not actually persist the write as expected (e.g. LocalFileStore) Another example: `s.popitem()` will pop a `(k, v)` pair off of the `s` store. That is, retrieve the `v` for `k`, delete the entry for `k`, and return a `(k, v)`. Note that unlike modern dicts which will return the last item that was stored (that is, LIFO (last-in, first-out) order), for KvPersisters there’s no assurance as to what item will be, since it will depend on the backend storage system and/or how the persister was implemented. #### clear() The clear method is disabled to make dangerous difficult. You don’t want to delete your whole DB If you really want to delete all your data, you can do so by doing something like this: ```python for k in self: del self[k] ``` or (in some cases) ```python for k in self: try: del self[k] except KeyError: pass ``` ### *class* dol.base.KvReader Bases: [`MappingViewMixin`](_autosummary/dol.base.html.md#dol.base.MappingViewMixin), [`Collection`](_autosummary/dol.base.html.md#dol.base.Collection), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) Acts as a Mapping abc, but with default \_\_len_\_ (implemented by counting keys) and head method to get the first (k, v) item of the store #### head() Get the first (key, value) pair ### dol.base.KvStore alias of [`Store`](_autosummary/dol.base.html.md#dol.base.Store) ### *class* dol.base.MappingViewMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Make `keys()`, `values()` and `items()` build their views from the `KeysView`, `ValuesView` and `ItemsView` class attributes, so a subclass can swap in its own view classes. #### *class* ItemsView(mapping) Bases: [`MappingView`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MappingView), [`Set`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Set) #### *class* KeysView(mapping) Bases: [`MappingView`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MappingView), [`Set`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Set) ### *class* dol.base.NoSuchItem Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Sentinel type; `no_such_item` is its instance. ### dol.base.Persister alias of [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister) ### dol.base.Reader alias of [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) ### *class* dol.base.Store(store=) Bases: [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister) By store we mean key-value store. This could be files in a filesystem, objects in s3, or a database. Where and how the content is stored should be specified, but StoreInterface offers a dict-like interface to this. ```default __getitem__ calls: _id_of_key _obj_of_data __setitem__ calls: _id_of_key _data_of_obj __delitem__ calls: _id_of_key __iter__ calls: _key_of_id ``` ```pycon >>> # Default store: no key or value conversion ##################################### >>> from dol import Store >>> s = Store() >>> s['foo'] = 33 >>> s['bar'] = 65 >>> assert list(s.items()) == [('foo', 33), ('bar', 65)] >>> assert list(s.store.items()) == [('foo', 33), ('bar', 65)] # see that the store contains the same thing >>> >>> ################################################################################# >>> # Now let's make stores that have a key and value conversion layer ############## >>> # input keys will be upper cased, and output keys lower cased ################### >>> # input values (assumed int) will be converted to ascii string, and visa versa ## >>> ################################################################################# >>> >>> def test_store(s): ... s['foo'] = 33 # write 33 to 'foo' ... assert 'foo' in s # __contains__ works ... assert 'no_such_key' not in s # __nin__ works ... s['bar'] = 65 # write 65 to 'bar' ... assert len(s) == 2 # there are indeed two elements ... assert list(s) == ['foo', 'bar'] # these are the keys ... assert list(s.keys()) == ['foo', 'bar'] # the keys() method works! ... assert list(s.values()) == [33, 65] # the values() method works! ... assert list(s.items()) == [('foo', 33), ('bar', 65)] # these are the items ... assert list(s.store.items()) == [('FOO', '!'), ('BAR', 'A')] # but note the internal representation ... assert s.get('foo') == 33 # the get method works ... assert s.get('no_such_key', 'something') == 'something' # return a default value ... del(s['foo']) # you can delete an item given its key ... assert len(s) == 1 # see, only one item left! ... assert list(s.items()) == [('bar', 65)] # here it is >>> >>> # We can introduce this conversion layer in several ways. Here's a few... ###################### >>> # by subclassing ############################################################################### >>> class MyStore(Store): ... def _id_of_key(self, k): ... return k.upper() ... def _key_of_id(self, _id): ... return _id.lower() ... def _data_of_obj(self, obj): ... return chr(obj) ... def _obj_of_data(self, data): ... return ord(data) >>> s = MyStore(store=dict()) # note that you don't need to specify dict(), since it's the default >>> test_store(s) >>> >>> # by assigning functions to converters ########################################################## >>> class MyStore(Store): ... def __init__(self, store, _id_of_key, _key_of_id, _data_of_obj, _obj_of_data): ... super().__init__(store) ... self._id_of_key = _id_of_key ... self._key_of_id = _key_of_id ... self._data_of_obj = _data_of_obj ... self._obj_of_data = _obj_of_data ... >>> s = MyStore(dict(), ... _id_of_key=lambda k: k.upper(), ... _key_of_id=lambda _id: _id.lower(), ... _data_of_obj=lambda obj: chr(obj), ... _obj_of_data=lambda data: ord(data)) >>> test_store(s) >>> >>> # using a Mixin class ############################################################################# >>> class Mixin: ... def _id_of_key(self, k): ... return k.upper() ... def _key_of_id(self, _id): ... return _id.lower() ... def _data_of_obj(self, obj): ... return chr(obj) ... def _obj_of_data(self, data): ... return ord(data) ... >>> class MyStore(Mixin, Store): # note that the Mixin must come before Store in the mro ... pass ... >>> s = MyStore() # no dict()? No, because default anyway >>> test_store(s) >>> >>> # adding wrapper methods to an already made Store instance ######################################### >>> s = Store(dict()) >>> s._id_of_key=lambda k: k.upper() >>> s._key_of_id=lambda _id: _id.lower() >>> s._data_of_obj=lambda obj: chr(obj) >>> s._obj_of_data=lambda data: ord(data) >>> test_store(s) ``` Note on defining your own “Mapping Views”. When you do a `.keys()`, a `.values()` or `.items()` you’re getting a `MappingView` instance; an iterable and sized container that provides some methods to access particular aspects of the wrapped mapping. If you need to customize the behavior of these instances, you should avoid overriding the `keys`, `values` or `items` methods directly, but instead override the `KeysView`, `ValuesView` or `ItemsView` classes that they use. For more, see: [https://github.com/i2mint/dol/wiki/Mapping-Views](https://github.com/i2mint/dol/wiki/Mapping-Views) #### get(k, default=None) * **Return type:** [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Val`) #### head() Get the first (key, value) pair * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) #### *classmethod* wrap(obj, class_trans=None, , delegation_attr='store') Wrap a `obj` (type or instance) with `delegator`. If obj is not a type, trivially returns `delegator(obj)`. The interesting case of `delegator_wrap` is when `obj` is a type (a class). In this case, `delegator_wrap` returns a callable (class or function) that has the same signature as obj, but that produces instances that are wrapped by `delegator` * **Parameters:** * **delegator** – An instance wrapper. A Callable (type or function – with only one required input) that will return a wrapped version of it’s input instance. * **obj** – The object (class or instance) to be wrapped. * **Returns:** A wrapped object Let’s demo this on a simple Delegator class. ```pycon >>> class Delegator: ... i_think = 'therefore I am delegated' # this is just to verify that we're in a Delegator ... def __init__(self, wrapped_obj): ... self.wrapped_obj = wrapped_obj ... def __getattr__(self, attr): # delegation: just forward attributes to wrapped_obj ... return getattr(self.wrapped_obj, attr) ... wrap = classmethod(delegator_wrap) # this is a useful recipe to have the Delegator carry it's own wrapping method ``` The only difference between a wrapped object `Delegator(obj)` and the original `obj` is that the wrapped one has a `i_think` attribute. The wrapped object should otherwise behave the same (on all but special (dunder) methods). So let’s test this on dictionaries, using the following test function: ```pycon >>> def test_wrapped_d(wrapped_d, original_d): ... '''A function to test a wrapped dict''' ... assert not hasattr(original_d, 'i_think') # verify that the unwrapped_d doesn't have an i_think attribute ... assert list(wrapped_d.items()) == list(original_d.items()) # verify that wrapped_d has an items that gives us the same thing as origina_d ... assert hasattr(wrapped_d, 'i_think') # ... but wrapped_d has a i_think attribute ... assert wrapped_d.i_think == 'therefore I am delegated' # ... and its what we set it to be ``` Let’s try delegating a dict INSTANCE first: ```pycon >>> d = {'a': 1, 'b': 2} >>> wrapped_d = delegator_wrap(Delegator, d) >>> test_wrapped_d(wrapped_d, d) ``` If we ask `delegator_wrap` to wrap a `dict` type, we get a subclass of Delegator (NOT dict!) whose instances will have the behavior exhibited above: ```pycon >>> WrappedDict = delegator_wrap(Delegator, dict, delegation_attr='wrapped_obj') >>> assert issubclass(WrappedDict, Delegator) >>> wrapped_d = WrappedDict(a=1, b=2) ``` ```pycon >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) ``` Now we’ll demo/test the `wrap = classmethod(delegator_wrap)` trick … with instances ```pycon >>> wrapped_d = Delegator.wrap(d) >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) ``` … with classes ```pycon >>> WrappedDict = Delegator.wrap(dict, delegation_attr='wrapped_obj') >>> wrapped_d = WrappedDict(a=1, b=2) ``` ```pycon >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) >>> class A(dict): ... def foo(self, x): ... pass >>> hasattr(A, 'foo') True >>> WrappedA = Delegator.wrap(A) >>> hasattr(WrappedA, 'foo') True ``` ### *class* dol.base.Stream(stream) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A layer-able version of the stream interface > \_\_iter_\_ calls: \_obj_of_data(map) ```pycon >>> from io import StringIO >>> >>> src = StringIO( ... '''a, b, c ... 1,2, 3 ... 4, 5,6 ... ''' ... ) >>> >>> from dol.base import Stream >>> >>> class MyStream(Stream): ... def _obj_of_data(self, line): ... return [x.strip() for x in line.strip().split(',')] ... >>> stream = MyStream(src) >>> >>> list(stream) [['a', 'b', 'c'], ['1', '2', '3'], ['4', '5', '6']] >>> stream.seek(0) # oh!... but we consumed the stream already, so let's go back to the beginning 0 >>> list(stream) [['a', 'b', 'c'], ['1', '2', '3'], ['4', '5', '6']] >>> stream.seek(0) # reverse again 0 >>> next(stream) ['a', 'b', 'c'] >>> next(stream) ['1', '2', '3'] ``` Let’s add a filter! There’s two kinds you can use. One that is applied to the line before the data is transformed by \_obj_of_data, and the other that is applied after (to the obj). ```pycon >>> from dol.base import Stream >>> from io import StringIO >>> >>> src = StringIO( ... '''a, b, c ... 1,2, 3 ... 4, 5,6 ... ''') >>> class MyFilteredStream(MyStream): ... def _post_filt(self, obj): ... return str.isnumeric(obj[0]) >>> >>> s = MyFilteredStream(src) >>> >>> list(s) [['1', '2', '3'], ['4', '5', '6']] >>> s.seek(0) 0 >>> list(s) [['1', '2', '3'], ['4', '5', '6']] >>> s.seek(0) 0 >>> next(s) ['1', '2', '3'] ``` Recipes: #### *classmethod* wrap(obj, class_trans=None, , delegation_attr='stream') Wrap a `obj` (type or instance) with `delegator`. If obj is not a type, trivially returns `delegator(obj)`. The interesting case of `delegator_wrap` is when `obj` is a type (a class). In this case, `delegator_wrap` returns a callable (class or function) that has the same signature as obj, but that produces instances that are wrapped by `delegator` * **Parameters:** * **delegator** – An instance wrapper. A Callable (type or function – with only one required input) that will return a wrapped version of it’s input instance. * **obj** – The object (class or instance) to be wrapped. * **Returns:** A wrapped object Let’s demo this on a simple Delegator class. ```pycon >>> class Delegator: ... i_think = 'therefore I am delegated' # this is just to verify that we're in a Delegator ... def __init__(self, wrapped_obj): ... self.wrapped_obj = wrapped_obj ... def __getattr__(self, attr): # delegation: just forward attributes to wrapped_obj ... return getattr(self.wrapped_obj, attr) ... wrap = classmethod(delegator_wrap) # this is a useful recipe to have the Delegator carry it's own wrapping method ``` The only difference between a wrapped object `Delegator(obj)` and the original `obj` is that the wrapped one has a `i_think` attribute. The wrapped object should otherwise behave the same (on all but special (dunder) methods). So let’s test this on dictionaries, using the following test function: ```pycon >>> def test_wrapped_d(wrapped_d, original_d): ... '''A function to test a wrapped dict''' ... assert not hasattr(original_d, 'i_think') # verify that the unwrapped_d doesn't have an i_think attribute ... assert list(wrapped_d.items()) == list(original_d.items()) # verify that wrapped_d has an items that gives us the same thing as origina_d ... assert hasattr(wrapped_d, 'i_think') # ... but wrapped_d has a i_think attribute ... assert wrapped_d.i_think == 'therefore I am delegated' # ... and its what we set it to be ``` Let’s try delegating a dict INSTANCE first: ```pycon >>> d = {'a': 1, 'b': 2} >>> wrapped_d = delegator_wrap(Delegator, d) >>> test_wrapped_d(wrapped_d, d) ``` If we ask `delegator_wrap` to wrap a `dict` type, we get a subclass of Delegator (NOT dict!) whose instances will have the behavior exhibited above: ```pycon >>> WrappedDict = delegator_wrap(Delegator, dict, delegation_attr='wrapped_obj') >>> assert issubclass(WrappedDict, Delegator) >>> wrapped_d = WrappedDict(a=1, b=2) ``` ```pycon >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) ``` Now we’ll demo/test the `wrap = classmethod(delegator_wrap)` trick … with instances ```pycon >>> wrapped_d = Delegator.wrap(d) >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) ``` … with classes ```pycon >>> WrappedDict = Delegator.wrap(dict, delegation_attr='wrapped_obj') >>> wrapped_d = WrappedDict(a=1, b=2) ``` ```pycon >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) >>> class A(dict): ... def foo(self, x): ... pass >>> hasattr(A, 'foo') True >>> WrappedA = Delegator.wrap(A) >>> hasattr(WrappedA, 'foo') True ``` ### dol.base.asis(p, k, v) Return `(p, k, v)` as is (the default `kv_walk` `leaf_yield`). * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### dol.base.delegate_to(wrapped, class_trans=None, delegation_attr='store', include=frozenset({}), ignore=frozenset({})) Class decorator factory: the decorated wrapper class constructs a `wrapped` instance and delegates to it, through `delegation_attr`, the attributes of `wrapped` (`dir(wrapped)` minus `ignore`, plus `include`) not already defined on the wrapper. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### dol.base.delegator_wrap(delegator, obj, class_trans=None, delegation_attr='store') Wrap a `obj` (type or instance) with `delegator`. If obj is not a type, trivially returns `delegator(obj)`. The interesting case of `delegator_wrap` is when `obj` is a type (a class). In this case, `delegator_wrap` returns a callable (class or function) that has the same signature as obj, but that produces instances that are wrapped by `delegator` * **Parameters:** * **delegator** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – An instance wrapper. A Callable (type or function – with only one required input) that will return a wrapped version of it’s input instance. * **obj** ([`type`](https://docs.python.org/3/builtins/functions.html#type) | [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The object (class or instance) to be wrapped. * **Returns:** A wrapped object Let’s demo this on a simple Delegator class. ```pycon >>> class Delegator: ... i_think = 'therefore I am delegated' # this is just to verify that we're in a Delegator ... def __init__(self, wrapped_obj): ... self.wrapped_obj = wrapped_obj ... def __getattr__(self, attr): # delegation: just forward attributes to wrapped_obj ... return getattr(self.wrapped_obj, attr) ... wrap = classmethod(delegator_wrap) # this is a useful recipe to have the Delegator carry it's own wrapping method ``` The only difference between a wrapped object `Delegator(obj)` and the original `obj` is that the wrapped one has a `i_think` attribute. The wrapped object should otherwise behave the same (on all but special (dunder) methods). So let’s test this on dictionaries, using the following test function: ```pycon >>> def test_wrapped_d(wrapped_d, original_d): ... '''A function to test a wrapped dict''' ... assert not hasattr(original_d, 'i_think') # verify that the unwrapped_d doesn't have an i_think attribute ... assert list(wrapped_d.items()) == list(original_d.items()) # verify that wrapped_d has an items that gives us the same thing as origina_d ... assert hasattr(wrapped_d, 'i_think') # ... but wrapped_d has a i_think attribute ... assert wrapped_d.i_think == 'therefore I am delegated' # ... and its what we set it to be ``` Let’s try delegating a dict INSTANCE first: ```pycon >>> d = {'a': 1, 'b': 2} >>> wrapped_d = delegator_wrap(Delegator, d) >>> test_wrapped_d(wrapped_d, d) ``` If we ask `delegator_wrap` to wrap a `dict` type, we get a subclass of Delegator (NOT dict!) whose instances will have the behavior exhibited above: ```pycon >>> WrappedDict = delegator_wrap(Delegator, dict, delegation_attr='wrapped_obj') >>> assert issubclass(WrappedDict, Delegator) >>> wrapped_d = WrappedDict(a=1, b=2) ``` ```pycon >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) ``` Now we’ll demo/test the `wrap = classmethod(delegator_wrap)` trick … with instances ```pycon >>> wrapped_d = Delegator.wrap(d) >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) ``` … with classes ```pycon >>> WrappedDict = Delegator.wrap(dict, delegation_attr='wrapped_obj') >>> wrapped_d = WrappedDict(a=1, b=2) ``` ```pycon >>> test_wrapped_d(wrapped_d, wrapped_d.wrapped_obj) >>> class A(dict): ... def foo(self, x): ... pass >>> hasattr(A, 'foo') True >>> WrappedA = Delegator.wrap(A) >>> hasattr(WrappedA, 'foo') True ``` ### dol.base.has_kv_store_interface(o) Check if object has the KvStore interface (that is, has the kv wrapper methods * **Parameters:** **o** – object (class or instance) * **Returns:** True if kv has the four key (in/out) and value (in/out) transformation methods ### dol.base.kv_walk(v, leaf_yield=, walk_filt=, pkv_to_pv=, \*, branch_yield=None, breadth_first=False, p=()) Walks a nested structure of mappings, yielding stuff on the way. * **Parameters:** * **v** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – A nested structure of mappings * **leaf_yield** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – (pp, k, vv) -> Any, what you want to yield when you encounter a leaf node (as define by walk_filt resolving to False) * **walk_filt** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – (p, k, vv) -> (bool) whether to explore the nested structure v further * **pkv_to_pv** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)]]) – (p, k, v) -> (pp, vv) where pp is a form of p + k (update of the path with the new node k) and vv is the value that will be used by both walk_filt and leaf_yield * **p** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`)) – The path to v (used internally, mainly, to keep track of the path) * **breadth_first** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to perform breadth-first traversal (instead of the default depth-first traversal). * **branch_yield** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – (pp, k, vv) -> Any, optional yield function to yield before the recursive walk of a branch. This is useful if you want to yield something for every branch, not just the leaves. * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ```pycon >>> d = {'a': 1, 'b': {'c': 2, 'd': 3}} >>> list(kv_walk(d)) [(('a',), 'a', 1), (('b', 'c'), 'c', 2), (('b', 'd'), 'd', 3)] >>> list(kv_walk(d, lambda p, k, v: '.'.join(p))) ['a', 'b.c', 'b.d'] ``` The `walk_filt` argument allows you to control what values the walk encountered should be walked through. This also means that this function is what controls when to stop the recursive traversal of the tree, and yield an actual “leaf”. Say we want to get (path, values) items from a nested mapping/store based on a `levels` argument that determines what the desired values are. This can be done as follows: ```pycon >>> def mk_level_walk_filt(levels): ... return lambda p, k, v: len(p) < levels - 1 ... >>> def leveled_map_walk(m, levels): ... yield from kv_walk( ... m, ... leaf_yield=lambda p, k, v: (p, v), ... walk_filt=mk_level_walk_filt(levels) ... ) >>> m = { ... 'a': {'b': {'c': 42}}, ... 'aa': {'bb': {'cc': 'dragon_con'}} ... } >>> >>> assert ( ... list(leveled_map_walk(m, 3)) ... == [ ... (('a', 'b', 'c'), 42), ... (('aa', 'bb', 'cc'), 'dragon_con') ... ] ... ) >>> assert ( ... list(leveled_map_walk(m, 2)) ... == [ ... (('a', 'b'), {'c': 42}), ... (('aa', 'bb'), {'cc': 'dragon_con'}) ... ] ... ) >>> >>> assert ( ... list(leveled_map_walk(m, 1)) ... == [ ... (('a',), {'b': {'c': 42}}), ... (('aa',), {'bb': {'cc': 'dragon_con'}}) ... ] ... ) ``` #### TIP If you want to use `kv_filt` to search and extract stuff from a nested mapping, you can have your `leaf_yield` return a sentinel (say, `None`) to indicate that the value should be skipped, and then filter out the `None` values from your results. ```pycon >>> mm = { ... 'a': {'b': {'c': 42}}, ... 'aa': {'bb': {'cc': 'meaning_of_life'}}, ... 'aaa': {'bbb': 314}, ... } >>> return_path_if_int_leaf = lambda p, k, v: (p, v) if isinstance(v, int) else None >>> list(filter(None, kv_walk(mm, leaf_yield=return_path_if_int_leaf))) [(('a', 'b', 'c'), 42), (('aaa', 'bbb'), 314)] ``` This “path search” functionality is available as a function in the `recipes` module, as `search_paths`. One last thing. Let’s demonstrate the use of `branch_yield` and `breadth_first`. Consider the following dictionary: ```pycon >>> d = {'big': {'apple': 1}, 'deal': 3, 'apple': {'pie': 1, 'crumble': 2}} ``` Say you wanted to find all the paths that end with ‘apple’. You could do: ```pycon >>> from functools import partial >>> yield_path_if_ends_with_apple = lambda p, k, v: p if k == 'apple' else None >>> walker1 = partial(kv_walk, leaf_yield=yield_path_if_ends_with_apple) >>> list(filter(None, walker1(d))) [('big', 'apple')] ``` It only got `('big', 'apple')` because the `leaf_yield` is only triggered for leaf nodes (as defined by the `walk_filt` argument, which defaults to `val_is_mapping`). So let’s try again, but this time, we’ll use `branch_yield` to yield the path for every branch (not just the leaves): ```pycon >>> walker2 = partial(walker1, branch_yield=yield_path_if_ends_with_apple) >>> list(filter(None, walker2(d))) [('big', 'apple'), ('apple',)] ``` But this isn’t convenient if you’d like your search to finish as soon as you find a path ending with `'apple'`. The order here comes from the fact that `kv_walk` does a depth-first traversal. If you want to do a breadth-first traversal, just say it: ```pycon >>> walker3 = partial(walker2, breadth_first=True) >>> list(filter(None, walker3(d))) [('apple',), ('big', 'apple')] ``` So now, you can get the first apple path by doing: ```pycon >>> next(filter(None, walker3(d))) ('apple',) ``` ### *class* dol.base.stream_util Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Small callbacks for `Stream`: an always-true filter, a no-op, and rewind (`skip_lines` currently only rewinds). ### dol.base.tuple_keypath_and_val(p, k, v) Extend the path `p` with the key `k` and return `(new_path, v)` (the default `kv_walk` `pkv_to_pv`). * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)] ### dol.base.val_is_mapping(p, k, v) Whether the walked value `v` is a `Mapping` (a `kv_walk` `walk_filt`). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### dol.base.wrapped_self(obj) Return the outermost transform-applying store wrapping `obj`, else `obj` itself. Use this inside a method DEFINED ON a class that was wrapped by dol’s delegation machinery (`wrap_kvs` / `filt_iter` / `cached_keys` / `mk_relative_path_store` / `Store.wrap` applied to the *class*). There, `self` is bound to the inner, unwrapped store, so `self[k]` bypasses the transforms (Issue #18). Writing `wrapped_self(self)[k]` recovers the outer, transform-applying view. On a direct `Store`/`KvReader` subclass, or a plain object that never went through the delegation machinery, there is no registered wrapper and `obj` is returned unchanged – a safe no-op. If the store was wrapped in several layers (e.g. via `Pipe`), this climbs to the OUTERMOST wrapper. Limitation: if the *same inner store instance* is wrapped by several live wrappers with different transforms (e.g. calling `wrap_kvs(shared_instance, ...)` twice), a method bound to that shared inner cannot know which wrapper it was reached through, so one of the wrappers is returned (ambiguous but never raw/None). This does not arise for class-wraps (each construction builds its own inner) nor for `copy.copy` (whose copies are transform-equivalent, so either answer is correct). * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ```pycon >>> import math >>> from dol import wrap_kvs, wrapped_self >>> sq = wrap_kvs(data_of_obj=lambda x: x * x, obj_of_data=lambda x: math.sqrt(x)) >>> @sq ... class S(dict): ... def via_self(self, k): ... return self[k] # self is the INNER store: NOT transformed ... def via_wrapped_self(self, k): ... return wrapped_self(self)[k] # outer store: transform applied >>> s = S() >>> s['2'] = 2 >>> s['2'] # external access is transformed 2.0 >>> s.via_self('2') # Issue #18: bypasses the transform 4 >>> s.via_wrapped_self('2') # wrapped_self recovers the transformed value 2.0 ``` # _autosummary/dol.caching.html.md # dol.caching Tools to add caching layers to stores and methods. This module provides comprehensive caching functionality for Python applications, offering flexible and powerful caching solutions for both data stores and method calls. Main Use Cases: - Property caching: Cache expensive computations that only need to be run once - Method caching: Cache method results based on arguments, with smart key generation - Store caching: Add caching layers to data stores for improved performance - Custom caching strategies: Flexible key generation and cache storage options Key Tools: - `cache_this`: The main decorator for caching properties and methods. Automatically detects whether to use property or method caching based on function signature. Supports custom cache storage, key functions, parameter ignoring, and serialization hooks. - `CachedProperty`: A descriptor for caching property values with flexible cache storage and key generation strategies. - `CachedMethod`: A descriptor for caching method results based on arguments, with support for parameter filtering and custom key functions. - `KeyStrategy` protocol: Extensible system for defining how cache keys are generated, including strategies for explicit keys, instance properties, method arguments, and composite keys. - Store decorators: Tools like `cache_vals`, `mk_sourced_store`, and `store_cached` for adding caching layers to data stores. ### Examples Basic property caching: ```pycon >>> class MyClass: ... @cache_this ... def expensive_computation(self): ... return sum(range(1000000)) ``` Method caching with argument-based keys: ```pycon >>> class Calculator: ... @cache_this(cache={}) ... def multiply(self, x, y): ... return x * y ``` Custom cache storage and key functions: ```pycon >>> class DataProcessor: ... def __init__(self): ... self.cache = {} ... @cache_this(cache='cache', ignore={'verbose'}) ... def process(self, data, mode='fast', verbose=False): ... return len(data) if mode == 'fast' else sum(data) ``` ### Functions | [`add_extension`](_autosummary/dol.caching.html.md#dol.caching.add_extension)([ext, name]) | Add an extension to a name. | |------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`cache_func_outputs`](_autosummary/dol.caching.html.md#dol.caching.cache_func_outputs)([cache]) | Decorator factory that caches a function's outputs in `cache`, keyed by `(func, args, kwargs)`. | | [`cache_property_method`](_autosummary/dol.caching.html.md#dol.caching.cache_property_method)([cls, method_name, ...]) | Converts a method of a class into a CachedProperty. | | [`cache_this`](_autosummary/dol.caching.html.md#dol.caching.cache_this)([func, cache, key, pre_cache, ...]) | Unified caching decorator for properties and methods with persistent storage support. | | [`cache_vals`](_autosummary/dol.caching.html.md#dol.caching.cache_vals)([store, cache, \_\_module_\_, ...]) | | | [`cached_method`](_autosummary/dol.caching.html.md#dol.caching.cached_method)([func, maxsize, typed]) | A decorator to cache the result of a method, ignoring the first argument (usually `self`). | | [`ensure_clear_to_kv_store`](_autosummary/dol.caching.html.md#dol.caching.ensure_clear_to_kv_store)(store) | Ensures the store has a working clear method. | | [`flush_on_exit`](_autosummary/dol.caching.html.md#dol.caching.flush_on_exit)(cls) | Class decorator: a subclass whose `__exit__` calls `flush_cache()` (adding a trivial `__enter__` if the class has none), so a write-cached store can be used as a context manager that flushes on exit. | | [`get_cache`](_autosummary/dol.caching.html.md#dol.caching.get_cache)(cache) | Convenience function to get a cache (whether it's already an instance, or needs to be validated). | | [`identity`](_autosummary/dol.caching.html.md#dol.caching.identity)(x) | Identity function that returns its input unchanged. | | [`is_a_cache`](_autosummary/dol.caching.html.md#dol.caching.is_a_cache)(obj) | Check if an object implements the cache interface. | | [`lru_cache_method`](_autosummary/dol.caching.html.md#dol.caching.lru_cache_method)([func, maxsize, typed]) | A decorator to cache the result of a method, ignoring the first argument (usually `self`). | | [`mk_cached_store`](_autosummary/dol.caching.html.md#dol.caching.mk_cached_store)([store, cache, \_\_module_\_, ...]) | | | [`mk_memoizer`](_autosummary/dol.caching.html.md#dol.caching.mk_memoizer)(cache) | Make a memoizer that caches the output of a getter function in a cache. | | [`mk_sourced_store`](_autosummary/dol.caching.html.md#dol.caching.mk_sourced_store)([store, source, ...]) | | | [`mk_write_cached_store`](_autosummary/dol.caching.html.md#dol.caching.mk_write_cached_store)([store, w_cache, ...]) | Wrap a write cache around a store. | | [`register_key_strategy`](_autosummary/dol.caching.html.md#dol.caching.register_key_strategy)(cls) | Register a class as a KeyStrategy. | | [`store_cached`](_autosummary/dol.caching.html.md#dol.caching.store_cached)(store, key_func) | Function output memorizer but using a specific (usually persisting) store as it's memory and a key_func to compute the key under which to store the output. | | [`store_cached_with_single_key`](_autosummary/dol.caching.html.md#dol.caching.store_cached_with_single_key)(store, key) | Function output memorizer but using a specific store and key as its memory. | ### Classes | [`ApplyToInstance`](_autosummary/dol.caching.html.md#dol.caching.ApplyToInstance)(func) | Apply a function to the instance to generate the key. | |------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------| | [`ApplyToMethodName`](_autosummary/dol.caching.html.md#dol.caching.ApplyToMethodName)(func) | Apply a function to the method name to generate the key. | | [`CachedMethod`](_autosummary/dol.caching.html.md#dol.caching.CachedMethod)(func[, cache, key, ignore, ...]) | Descriptor that caches the result of method calls based on their arguments. | | [`CachedProperty`](_autosummary/dol.caching.html.md#dol.caching.CachedProperty)(func[, cache, key, ...]) | Descriptor that caches the result of the first call to a method. | | [`CompositeKey`](_autosummary/dol.caching.html.md#dol.caching.CompositeKey)(\*strategies[, separator]) | Combine multiple key strategies into a single composite key. | | [`ExplicitKey`](_autosummary/dol.caching.html.md#dol.caching.ExplicitKey)(key) | Use an explicitly provided key value. | | [`FromMethodArgs`](_autosummary/dol.caching.html.md#dol.caching.FromMethodArgs)(func) | Apply a function to method arguments to generate the key. | | [`HashableDict`](_autosummary/dol.caching.html.md#dol.caching.HashableDict) | Just a dict, but hashable | | [`HashableMixin`](_autosummary/dol.caching.html.md#dol.caching.HashableMixin)() | Mixin making instances hashable by identity (`id(self)`). | | [`InstanceProp`](_autosummary/dol.caching.html.md#dol.caching.InstanceProp)(prop_name) | Get a key from an instance property. | | [`KeyStrategy`](_autosummary/dol.caching.html.md#dol.caching.KeyStrategy)(\*args, \*\*kwargs) | Protocol defining how a key strategy should behave. | | [`WriteBackChainMap`](_autosummary/dol.caching.html.md#dol.caching.WriteBackChainMap)(\*maps) | A collections.ChainMap that also 'writes back' when a key is found. | ### *class* dol.caching.ApplyToInstance(func) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Apply a function to the instance to generate the key. #### resolve_at_definition(method_name) Cannot resolve at definition time, need the instance. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### resolve_at_runtime(instance, method_name) Apply the function to the instance at runtime. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### *class* dol.caching.ApplyToMethodName(func) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Apply a function to the method name to generate the key. ```pycon >>> strategy = ApplyToMethodName(lambda name: f"{name}.cache") >>> strategy.resolve_at_definition("my_method") 'my_method.cache' ``` #### resolve_at_definition(method_name) Apply the function to the method name at definition time. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### *class* dol.caching.CachedMethod(func, cache=None, key=None, \*, ignore=None, allow_none_keys=False, lock_factory=, pre_cache=False, serialize=None, deserialize=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Descriptor that caches the result of method calls based on their arguments. Similar to CachedProperty but handles methods with arguments, caching results based on unique combinations of arguments (excluding self). ### *class* dol.caching.CachedProperty(func, cache=None, key=None, \*, allow_none_keys=False, lock_factory=, pre_cache=False, serialize=None, deserialize=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Descriptor that caches the result of the first call to a method. It generalizes the builtin functools.cached_property class, enabling the user to specify a cache object and a key to store the cache value. ### *class* dol.caching.CompositeKey(\*strategies, separator='_') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Combine multiple key strategies into a single composite key. Useful for creating keys that depend on both instance properties and method arguments. #### resolve_at_definition(method_name) Try to resolve all strategies at definition time. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) | [`None`](https://docs.python.org/3/builtins/constants.html#None) #### resolve_at_runtime(instance, method_name, \*args, \*\*kwargs) Resolve all strategies at runtime and combine them. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### *class* dol.caching.ExplicitKey(key) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Use an explicitly provided key value. ```pycon >>> strategy = ExplicitKey("my_key") >>> strategy.resolve_at_definition("method_name") 'my_key' ``` #### resolve_at_definition(method_name) Return the explicit key value at definition time. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### *class* dol.caching.FromMethodArgs(func) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Apply a function to method arguments to generate the key. The function receives `(self, *args, **kwargs)` and should return a cache key. #### resolve_at_definition(method_name) Cannot resolve at definition time, need the arguments. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### resolve_at_runtime(instance, method_name, \*args, \*\*kwargs) Apply the function to the instance and method arguments at runtime. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### *class* dol.caching.HashableDict Bases: [`HashableMixin`](_autosummary/dol.caching.html.md#dol.caching.HashableMixin), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) Just a dict, but hashable ### *class* dol.caching.HashableMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mixin making instances hashable by identity (`id(self)`). ### *class* dol.caching.InstanceProp(prop_name) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Get a key from an instance property. #### resolve_at_definition(method_name) Cannot resolve at definition time, need the instance. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### resolve_at_runtime(instance, method_name) Get the property value from the instance at runtime. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### *class* dol.caching.KeyStrategy(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Protocol defining how a key strategy should behave. #### resolve_at_definition(method_name) Attempt to resolve the key at class definition time. * **Parameters:** **method_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the method being decorated. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) | [`None`](https://docs.python.org/3/builtins/constants.html#None) * **Returns:** The resolved key or None if it can’t be resolved at definition time. #### resolve_at_runtime(instance, method_name) Resolve the key at runtime. By default, this will call resolve_at_definition on method_name. * **Parameters:** * **instance** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The instance the property is being accessed on. * **method_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the method being decorated. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) * **Returns:** The resolved key. ### *class* dol.caching.WriteBackChainMap(\*maps) Bases: [`ChainMap`](https://docs.python.org/3/library/collections.html#collections.ChainMap) A collections.ChainMap that also ‘writes back’ when a key is found. ```pycon >>> from dol.caching import WriteBackChainMap >>> >>> d = WriteBackChainMap({'a': 1, 'b': 2}, {'b': 22, 'c': 33}, {'d': 444}) ``` In a `ChainMap`, when you ask for the value for a key, each mapping in the sequence is checked for, and the first mapping found that contains it will be the one determining the value. So here if you look for `b`, though the first mapping will give you the value, though the second mapping also contains a `b` with a different value: ```pycon >>> d['b'] 2 ``` if you ask for `c`, it’s the second mapping that will give you the value: ```pycon >>> d['c'] 33 ``` But unlike with the builtin `ChainMap`, something else is going to happen here: ```pycon >>> d WriteBackChainMap({'a': 1, 'b': 2, 'c': 33}, {'b': 22, 'c': 33}, {'d': 444}) ``` See that now the first mapping also has the `('c', 33)` key-value pair: That is what we call “write back”. When a key is found in a mapping, all previous mappings (which by definition of `ChainMap` did not have a value for that key) will be revisited and that key-value pair will be written in it. As in with `ChainMap`, all writes will be carried out in the first mapping, and only the first mapping: ```pycon >>> d['e'] = 5 >>> d WriteBackChainMap({'a': 1, 'b': 2, 'c': 33, 'e': 5}, {'b': 22, 'c': 33}, {'d': 444}) ``` Example use cases: - You’re working with a local and a remote source of data. You’d like to list the keys available in both, and use the local item if it’s available, and if it’s not, you want it to be sourced from remote, but written in local for quicker access next time. - You have several sources to look for configuration values: a sequence of configuration files/folders to look through (like a unix search path for command resolution) and environment variables. ### dol.caching.add_extension(ext=None, name=None) Add an extension to a name. If name is None, return a partial function that will add the extension to a name when called. add_extension is a useful helper for making key functions, namely for cache_this. ```pycon >>> add_extension('txt', 'file') 'file.txt' >>> add_txt_ext = add_extension('txt') >>> add_txt_ext('file') 'file.txt' ``` #### NOTE If you want to add an extension to a name that already has an extension, you can do that, but it will add the extension to the end of the name, not replace the existing extension. ```pycon >>> add_txt_ext('file.txt') 'file.txt.txt' ``` Also, bare in mind that if ext starts with the system’s extension separator, (os.path.extsep), it will be removed. ```pycon >>> add_extension('.txt', 'file') == add_extension('txt', 'file') == 'file.txt' True ``` ### dol.caching.cache_func_outputs(cache=) Decorator factory that caches a function’s outputs in `cache`, keyed by `(func, args, kwargs)`. * **Parameters:** **cache** – A cache instance (with `__contains__`/`__getitem__`/ `__setitem__`), or a zero-arg factory/class (e.g. the default, `HashableDict`) used to make one. #### NOTE like `args`, every `kwargs` value must be hashable (they’re part of the cache key). Passing an unhashable value (e.g. a `list`) raises `TypeError` rather than silently skipping the cache. ```pycon >>> @cache_func_outputs() ... def f(x, y=2): ... print(f"computing f({x}, {y})") ... return x + y >>> f(1) computing f(1, 2) 3 >>> f(1) # cached: no "computing" print 3 >>> f(1, y=3) # different kwargs: not a cache hit computing f(1, 3) 4 >>> f(1, y=3) # this one is now cached too 4 ``` ### dol.caching.cache_property_method(cls=None, method_name=None, \*, cache_decorator=) Converts a method of a class into a CachedProperty. Essentially, it does what `A.method = cache_this(A.method)` would do, taking care of the `__set_name__` problem that you’d run into doing it that way. Note that here, you need to say `cache_property_method(A, 'method')`. * **Parameters:** * **cls** ([*type*](https://docs.python.org/3/builtins/functions.html#type)) – The class containing the method. * **method_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the method to convert to a cached property. * **cache_decorator** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – The decorator to use to cache the method. Defaults to `cache_this`. One frequent use case would be to use `functools.partial` to fix the cache and key parameters of `cache_this` and inject that. ### Example ```pycon >>> @cache_property_method(['normal_method', 'property_method']) ... class TestClass: ... def normal_method(self): ... print('normal_method called') ... return 1 ... ... @property ... def property_method(self): ... print('property_method called') ... return 2 >>> >>> c = TestClass() >>> c.normal_method normal_method called 1 >>> c.normal_method 1 >>> c.property_method property_method called 2 >>> c.property_method 2 ``` You can also use it like this: ```pycon >>> class TestClass: ... def normal_method(self): ... print('normal_method called') ... return 1 ... ... @property ... def property_method(self): ... print('property_method called') ... return 2 >>> >>> cache_property_method( ... TestClass, ... [ ... 'normal_method', ... 'property_method', ... ], ... ) >>> c = TestClass() >>> c.normal_method normal_method called 1 >>> c.normal_method 1 >>> c.property_method property_method called 2 >>> c.property_method 2 ``` ### dol.caching.cache_this(func=None, , cache=None, key=None, pre_cache=False, as_property=None, ignore=None, serialize=None, deserialize=None) Unified caching decorator for properties and methods with persistent storage support. `cache_this` extends the capabilities of Python’s built-in `functools.cached_property` and `functools.lru_cache` by providing: - **Persistent caching**: Store cached values in files, databases, or any MutableMapping - **Flexible cache backends**: Use instance attributes, external stores, or cache factories - **Smart key generation**: Automatic argument-based keys for methods with parameter filtering - **Serialization support**: Custom serialize/deserialize functions for complex data - **Auto-detection**: Automatically chooses property vs method caching based on signature - **No LRU eviction**: Unlike lru_cache, values persist until explicitly removed Unlike functools.cached_property (properties only) and lru_cache (memory-only with eviction), cache_this provides a unified interface for both use cases with persistent storage options. * **Parameters:** * **func** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)]) – The function to be decorated (usually left empty). * **cache** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – The cache storage. Can be: - A MutableMapping instance (shared across instances) - A string naming an instance attribute containing a MutableMapping - A callable taking (instance) and returning a MutableMapping This enables instance-specific caching, e.g.: cache=lambda self: Files(f’/cache/{self.user_id}/’) * **key** (`Union`[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – For properties: the key to store the cache value, can be a callable that will be applied to the method name to make a key, or an explicit string. For methods: a callable that takes `(self, *args, **kwargs)` and returns a cache key. * **pre_cache** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool) | [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)) – Default is False. If True, adds an in-memory cache to the method to (also) cache the results in memory. If a MutableMapping is given, it will be used as the pre-cache. This is useful when you want a persistent cache but also want to speed up access to the method in the same session. * **as_property** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – If True, force use of CachedProperty. If False, force use of CachedMethod. If None (default), auto-detect based on function signature. * **ignore** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Parameter name(s) to exclude from cache key computation. Can be a string (single parameter) or list of strings (multiple parameters). Commonly used to ignore ‘self’ or parameters like ‘verbose’ that don’t affect the result. * **serialize** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Optional function to serialize values before caching (e.g. `pickle.dumps` for binary file storage). * **deserialize** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Optional function to deserialize cached values (e.g. `pickle.loads`). * **Returns:** The decorated function. ### Comprehensive Example Here’s a complete example showcasing all major features of cache_this: ```pycon >>> import tempfile >>> import os >>> from pathlib import Path >>> >>> class DataProcessor: ... def __init__(self, user_id="user123"): ... self.user_id = user_id ... self.memory_cache = {} # In-memory cache ... self.call_counts = {} # Track function calls for demo ... ... # 1. Basic property caching (like functools.cached_property) ... @cache_this ... def basic_property(self): ... '''Cached in instance.__dict__ by default''' ... self.call_counts['basic_property'] = self.call_counts.get('basic_property', 0) + 1 ... return f"computed_value_{self.call_counts['basic_property']}" ... ... # 2. Property with custom cache and key ... @cache_this(cache='memory_cache', key='custom_prop_key') ... def custom_cached_property(self): ... '''Cached in instance.memory_cache with custom key''' ... self.call_counts['custom_cached_property'] = self.call_counts.get('custom_cached_property', 0) + 1 ... return f"custom_value_{self.call_counts['custom_cached_property']}" ... ... # 3. Method caching with argument-based keys ... @cache_this(cache='memory_cache') ... def compute_result(self, x, y, mode='fast'): ... '''Cached based on arguments (x, y, mode)''' ... key = ('compute_result', x, y, mode) ... self.call_counts[key] = self.call_counts.get(key, 0) + 1 ... return x * y * (2 if mode == 'fast' else 3) ... ... # 4. Method caching with ignored parameters ... @cache_this(cache='memory_cache', ignore={'verbose', 'debug'}) ... def process_data(self, data, algorithm='default', verbose=False, debug=False): ... '''Cache ignores verbose and debug parameters''' ... key = ('process_data', tuple(data), algorithm) ... self.call_counts[key] = self.call_counts.get(key, 0) + 1 ... if verbose: print(f"Processing {data} with {algorithm}") ... return sum(data) * (2 if algorithm == 'default' else 3) ... ... # 5. Instance-specific cache factory ... @cache_this(cache=lambda self: {f'{self.user_id}_cache': {}}.get(f'{self.user_id}_cache')) ... def user_specific_computation(self, value): ... '''Each instance gets its own cache based on user_id''' ... key = ('user_specific_computation', value) ... self.call_counts[key] = self.call_counts.get(key, 0) + 1 ... return value ** 2 ``` Now let’s test all the features: ```pycon >>> processor = DataProcessor("alice") >>> >>> # Test basic property caching >>> result1 = processor.basic_property >>> result2 = processor.basic_property # Should use cache >>> assert result1 == result2 == "computed_value_1" >>> assert 'basic_property' in processor.__dict__ # Cached in instance dict >>> >>> # Test custom cache and key >>> result1 = processor.custom_cached_property >>> result2 = processor.custom_cached_property # Should use cache >>> assert result1 == result2 == "custom_value_1" >>> assert 'custom_prop_key' in processor.memory_cache >>> >>> # Test method caching with arguments >>> result1 = processor.compute_result(3, 4, 'fast') >>> result2 = processor.compute_result(3, 4, 'fast') # Should use cache >>> result3 = processor.compute_result(3, 4, 'slow') # Different args, new computation >>> assert result1 == result2 == 24 # 3 * 4 * 2 >>> assert result3 == 36 # 3 * 4 * 3 >>> >>> # Test parameter ignoring >>> result1 = processor.process_data([1, 2, 3], verbose=True) Processing [1, 2, 3] with default >>> result2 = processor.process_data([1, 2, 3], verbose=False) # Should use same cache >>> result3 = processor.process_data([1, 2, 3], debug=True) # Should use same cache >>> assert result1 == result2 == result3 == 12 # sum([1,2,3]) * 2 >>> >>> # Test instance-specific caching >>> result1 = processor.user_specific_computation(5) >>> result2 = processor.user_specific_computation(5) # Should use cache >>> assert result1 == result2 == 25 # 5 ** 2 >>> >>> # Different instance should have separate cache >>> processor2 = DataProcessor("bob") >>> result3 = processor2.user_specific_computation(5) # Fresh computation >>> assert result3 == 25 ``` Used with no arguments, `cache_this` will cache just as the builtin `cached_property` does – in the instance’s `__dict__` attribute. ```pycon >>> class SameAsCachedProperty: ... @cache_this ... def foo(self): ... print("In SameAsCachedProperty.foo...") ... return 42 ... >>> obj = SameAsCachedProperty() >>> obj.__dict__ # the cache is empty {} >>> obj.foo # when we access foo, it's computed and returned... In SameAsCachedProperty.foo... 42 >>> obj.__dict__ # ... but also cached {'foo': 42} >>> obj.foo # so that the next time we access foo, it's returned from the cache. 42 ``` Not that if you specify `cache=False`, you get a property that is computed every time it’s accessed: ```pycon >>> class NoCache: ... @cache_this(cache=False) ... def foo(self): ... print("In NoCache.foo...") ... return 42 ... >>> obj = NoCache() >>> obj.foo In NoCache.foo... 42 >>> obj.foo In NoCache.foo... 42 ``` Specify the cache as a dictionary that lives outside the instance: ```pycon >>> external_cache = {} >>> >>> class CacheWithExternalMapping: ... @cache_this(cache=external_cache) ... def foo(self): ... print("In CacheWithExternalMapping.foo...") ... return 42 ... >>> obj = CacheWithExternalMapping() >>> external_cache {} >>> obj.foo In CacheWithExternalMapping.foo... 42 >>> external_cache {'foo': 42} >>> obj.foo 42 ``` Specify the cache as an attribute of the instance, and an explicit key: ```pycon >>> class WithCacheInInstanceAttribute: ... ... def __init__(self): ... self.my_cache = {} ... ... @cache_this(cache='my_cache', key='key_for_foo') ... def foo(self): ... print("In WithCacheInInstanceAttribute.foo...") ... return 42 ... >>> obj = WithCacheInInstanceAttribute() >>> obj.my_cache {} >>> obj.foo In WithCacheInInstanceAttribute.foo... 42 >>> obj.my_cache {'key_for_foo': 42} >>> obj.foo 42 ``` Now let’s see a more involved example that exhibits how `cache_this` would be used in real life. Note two things in the example below. First, that we use `functools.partial` to fix the parameters of our `cache_this`. This enables us to reuse the same `cache_this` in multiple places without all the verbosity. We fix that the cache is the attribute `cache` of the instance, and that the key is a function that will be computed from the name of the method adding a `'.pkl'` extension to it. Secondly, we use the `ValueCodecs` from `dol` to provide a pickle codec for storying values. The backend store used here is a dictionary, so we don’t really need a codec to store values, but in real life you would use a persistent storage that would require a codec, such as files or a database. Thirdly, we’ll use a `pre_cache` to store the values in a different cache “before” (setting and getting) them in the main cache. This is useful, for instance, when you want to persist the values (in the main cache), but keep them in memory for faster access in the same session (the pre-cache, a dict() instance usually). It can also be used to store and use things locally (pre-cache) while sharing them with others by storing them in a remote store (main cache). Finally, we’ll use a dict that logs any setting and getting of values to show how the caches are being used. ```pycon >>> from dol import cache_this >>> >>> from functools import partial >>> from dol import ValueCodecs >>> from collections import UserDict >>> >>> >>> class LoggedCache(UserDict): ... name = 'cache' ... ... def __setitem__(self, key, value): ... print(f"In {self.name}: setting {key} to {value}") ... return super().__setitem__(key, value) ... ... def __getitem__(self, key): ... print(f"In {self.name}: getting value of {key}") ... return super().__getitem__(key) ... >>> >>> class CacheA(LoggedCache): ... name = 'CacheA' ... >>> >>> class CacheB(LoggedCache): ... name = 'CacheB' ... >>> >>> cache_with_pickle = partial( ... cache_this, ... cache='cache', # the cache can be found on the instance attribute `cache` ... key=lambda x: f"{x}.pkl", # the key is the method name with a '.pkl' extension ... pre_cache=CacheB(), ... ) >>> >>> >>> class PickleCached: ... def __init__(self, backend_store_factory=CacheA): ... # usually this would be a mapping interface to persistent storage: ... self._backend_store = backend_store_factory() ... self.cache = ValueCodecs.default.pickle(self._backend_store) ... ... @cache_with_pickle ... def foo(self): ... print("In PickleCached.foo...") ... return 42 ... ``` ```pycon >>> obj = PickleCached() >>> list(obj.cache) [] ``` ```pycon >>> obj.foo In CacheA: getting value of foo.pkl In CacheA: getting value of foo.pkl In PickleCached.foo... In CacheA: setting foo.pkl to b'\x80\x04K*.' 42 >>> obj.foo In CacheA: getting value of foo.pkl In CacheB: setting foo.pkl to 42 42 ``` As usual, it’s because the cache now holds something that has to do with `foo`: ```pycon >>> list(obj.cache) ['foo.pkl'] >>> # == ['foo.pkl'] ``` The value of `'foo.pkl'` is indeed `42`: ```pycon >>> obj.cache['foo.pkl'] In CacheA: getting value of foo.pkl 42 ``` But note that the actual way it’s stored in the `_backend_store` is as pickle bytes: ```pycon >>> obj._backend_store['foo.pkl'] In CacheA: getting value of foo.pkl b'\x80\x04K*.' >>> # == b'\x80\x04K*.' ``` ### dol.caching.cache_vals(store=None, \*, cache=, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) * **Parameters:** * **store** – The class of the store you want to cache * **cache** – The store you want to use to cache. Anything with a \_\_setitem_\_(k, v) and a \_\_getitem_\_(k). By default, it will use a dict * **Returns:** A subclass of the input store, but with caching (to the cache store) ```pycon >>> from dol.caching import cache_vals >>> import time >>> class SlowDict(dict): ... sleep_s = 0.2 ... def __getitem__(self, k): ... time.sleep(self.sleep_s) ... return super().__getitem__(k) ... ... >>> d = SlowDict({'a': 1, 'b': 2, 'c': 3}) >>> >>> d['a'] # Wow! Takes a long time to get 'a' 1 >>> cache = dict() >>> CachedSlowDict = cache_vals(store=SlowDict, cache=cache) >>> >>> s = CachedSlowDict({'a': 1, 'b': 2, 'c': 3}) >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: [] >>> # This will take a LONG time because it's the first time we ask for 'a' >>> v = s['a'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a'] >>> # This will take very little time because we have 'a' in the cache >>> v = s['a'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a'] >>> # But we don't have 'b' >>> v = s['b'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a', 'b'] >>> # But now we have 'b' >>> v = s['b'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a', 'b'] >>> s['d'] = 4 # and we can do things normally (like put stuff in the store) >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c', 'd'] cache: ['a', 'b'] >>> s['d'] # if we ask for it again though, it will take time (the first time) 4 >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c', 'd'] cache: ['a', 'b', 'd'] >>> # Of course, we could write 'd' in the cache as well, to get it quicker, >>> # but that's another story: The story of write caches! >>> >>> # And by the way, your "cache wrapped" store hold a pointer to the cache it's using, >>> # so you can take a peep there if needed: >>> s._cache {'a': 1, 'b': 2, 'd': 4} ``` ### dol.caching.cached_method(func=None, , maxsize=128, typed=False) A decorator to cache the result of a method, ignoring the first argument (usually `self`). This decorator uses `functools.lru_cache` to cache the method result based on the arguments passed to the method, excluding the first argument (typically `self`). This allows methods of a class to be cached while ignoring the instance (`self`) in the cache key. * **Parameters:** * **func** – The method to be decorated. If not provided, a partially applied decorator will be returned for later application. * **maxsize** – The maximum size of the cache. * **typed** – If True, cache entries will be different based on argument types, such as distinguishing between `1` and `1.0`. * **Returns:** A wrapped function with LRU caching applied, ignoring the first argument (`self`). ### Example ```pycon >>> class MyClass: ... @cached_method(maxsize=2, typed=True) ... def add(self, x, y): ... print(f"Computing {x} + {y}") ... return x + y ... >>> obj = MyClass() >>> obj.add(1, 2) Computing 1 + 2 3 >>> obj.add(1, 2) # Cached result, no recomputation 3 >>> obj.add(1.0, 2.0) # Different types, recomputation occurs Computing 1.0 + 2.0 3.0 ``` ### dol.caching.ensure_clear_to_kv_store(store) Ensures the store has a working clear method. If the store doesn’t have a clear method or has the disabled version, adds a proper implementation that safely removes all items. * **Parameters:** **store** – A Store class or instance * **Returns:** The same store with guaranteed clear functionality ```pycon >>> class NoClearing(dict): ... clear = None >>> d = NoClearing({'a': 1, 'b': 2}) >>> d = ensure_clear_to_kv_store(d) >>> len(d) 2 >>> d.clear() >>> len(d) 0 ``` ### dol.caching.flush_on_exit(cls) Class decorator: a subclass whose `__exit__` calls `flush_cache()` (adding a trivial `__enter__` if the class has none), so a write-cached store can be used as a context manager that flushes on exit. Used by `mk_write_cached_store`. ### dol.caching.get_cache(cache) Convenience function to get a cache (whether it’s already an instance, or needs to be validated). ```pycon >>> get_cache({'a': 1}) # Return existing cache instance {'a': 1} >>> get_cache(dict)() # Return result of calling cache factory {} ``` ### dol.caching.identity(x) Identity function that returns its input unchanged. * **Return type:** [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`T`) ```pycon >>> identity(42) 42 >>> identity("hello") 'hello' >>> identity([1, 2, 3]) [1, 2, 3] ``` ### dol.caching.is_a_cache(obj) Check if an object implements the cache interface. A cache object must have \_\_contains_\_, \_\_getitem_\_, and \_\_setitem_\_ methods. ```pycon >>> is_a_cache({}) # dict is a valid cache True >>> is_a_cache([]) # list has these methods but for indexed access True >>> is_a_cache("string") # string is not (immutable) False ``` ### dol.caching.lru_cache_method(func=None, , maxsize=128, typed=False) A decorator to cache the result of a method, ignoring the first argument (usually `self`). This decorator uses `functools.lru_cache` to cache the method result based on the arguments passed to the method, excluding the first argument (typically `self`). This allows methods of a class to be cached while ignoring the instance (`self`) in the cache key. * **Parameters:** * **func** – The method to be decorated. If not provided, a partially applied decorator will be returned for later application. * **maxsize** – The maximum size of the cache. * **typed** – If True, cache entries will be different based on argument types, such as distinguishing between `1` and `1.0`. * **Returns:** A wrapped function with LRU caching applied, ignoring the first argument (`self`). ### Example ```pycon >>> class MyClass: ... @lru_cache_method ... def add(self, x, y): ... print(f"Computing {x} + {y}") ... return x + y >>> obj = MyClass() >>> obj.add(1, 2) Computing 1 + 2 3 >>> obj.add(1, 2) # Cached result, no recomputation 3 ``` Like `lru_cache`, you can specify the `maxsize` and `typed` parameters: ```pycon >>> class MyOtherClass: ... @lru_cache_method(maxsize=2, typed=True) ... def add(self, x, y): ... print(f"Computing {x} + {y}") ... return x + y ... >>> obj = MyOtherClass() >>> obj.add(1, 2) Computing 1 + 2 3 >>> obj.add(1, 2) # Cached result, no recomputation 3 >>> obj.add(1.0, 2.0) # Different types, recomputation occurs Computing 1.0 + 2.0 3.0 ``` ### dol.caching.mk_cached_store(store=None, \*, cache=, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) * **Parameters:** * **store** – The class of the store you want to cache * **cache** – The store you want to use to cache. Anything with a \_\_setitem_\_(k, v) and a \_\_getitem_\_(k). By default, it will use a dict * **Returns:** A subclass of the input store, but with caching (to the cache store) ```pycon >>> from dol.caching import cache_vals >>> import time >>> class SlowDict(dict): ... sleep_s = 0.2 ... def __getitem__(self, k): ... time.sleep(self.sleep_s) ... return super().__getitem__(k) ... ... >>> d = SlowDict({'a': 1, 'b': 2, 'c': 3}) >>> >>> d['a'] # Wow! Takes a long time to get 'a' 1 >>> cache = dict() >>> CachedSlowDict = cache_vals(store=SlowDict, cache=cache) >>> >>> s = CachedSlowDict({'a': 1, 'b': 2, 'c': 3}) >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: [] >>> # This will take a LONG time because it's the first time we ask for 'a' >>> v = s['a'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a'] >>> # This will take very little time because we have 'a' in the cache >>> v = s['a'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a'] >>> # But we don't have 'b' >>> v = s['b'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a', 'b'] >>> # But now we have 'b' >>> v = s['b'] >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c'] cache: ['a', 'b'] >>> s['d'] = 4 # and we can do things normally (like put stuff in the store) >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c', 'd'] cache: ['a', 'b'] >>> s['d'] # if we ask for it again though, it will take time (the first time) 4 >>> print(f"store: {list(s)}\ncache: {list(cache)}") store: ['a', 'b', 'c', 'd'] cache: ['a', 'b', 'd'] >>> # Of course, we could write 'd' in the cache as well, to get it quicker, >>> # but that's another story: The story of write caches! >>> >>> # And by the way, your "cache wrapped" store hold a pointer to the cache it's using, >>> # so you can take a peep there if needed: >>> s._cache {'a': 1, 'b': 2, 'd': 4} ``` ### dol.caching.mk_memoizer(cache) Make a memoizer that caches the output of a getter function in a cache. #### NOTE This is a specialized memoizer for getter functions/methods, i.e. functions/methods that have the signature (instance, key) and return a value. * **Parameters:** **cache** – The cache to use. Must have \_\_getitem_\_ and \_\_setitem_\_ methods. * **Returns:** A memoizer that caches the output of the function in the cache. ```pycon >>> cache = dict() >>> @mk_memoizer(cache) ... def getter(self, k): ... print(f"getting value for {k}...") ... return k * 10 ... >>> getter(None, 2) getting value for 2... 20 >>> getter(None, 2) 20 ``` ### dol.caching.mk_sourced_store(store=None, , source=None, return_source_data=True, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) * **Parameters:** * **store** – The class of the store you want to cache * **cache** – The store you want to use to cache. Anything with a \_\_setitem_\_(k, v) and a \_\_getitem_\_(k). By default, it will use a dict * **return_source_data** * **store** – The class of the store you’re talking to. This store acts as the cache * **source** – The store that is used to populate the store (cache) when a key is missing there. * **return_source_data** – If True, will return `source[k]` as is. This should be used only if `store[k]` would return the same. If False, will first write to cache (`store[k] = source[k]`) then return `store[k]`. The latter introduces a performance hit (we write and then read again from the cache), but ensures consistency (and is useful if the writing or the reading to/from store transforms the data in some way. * **Returns:** A subclass of the input store, but with caching (to the cache store) * **Returns:** A decorated store Here are two stores pretending to be local and remote data stores respectively. ```pycon >>> from dol.caching import mk_sourced_store >>> >>> class Local(dict): ... def __getitem__(self, k): ... print(f"looking for {k} in Local") ... return super().__getitem__(k) >>> >>> class Remote(dict): ... def __getitem__(self, k): ... print(f"looking for {k} in Remote") ... return super().__getitem__(k) ``` Let’s make a remote store with two elements in it, and a local store class that asks the remote store for stuff if it can’t find it locally. ```pycon >>> remote = Remote({'foo': 'bar', 'hello': 'world'}) >>> SourcedLocal = mk_sourced_store(Local, source=remote) >>> s = SourcedLocal({'some': 'local stuff'}) >>> list(s) # the local store has one key ['some'] ``` But if we ask for a key that is in the remote store, it provides it: ```pycon >>> assert s['foo'] == 'bar' looking for foo in Local looking for foo in Remote ``` ```pycon >>> list(s) ['some', 'foo'] ``` See that next time we ask for the ‘foo’ key, the local store provides it: ```pycon >>> assert s['foo'] == 'bar' looking for foo in Local ``` ```pycon >>> assert s['hello'] == 'world' looking for hello in Local looking for hello in Remote >>> list(s) ['some', 'foo', 'hello'] ``` We can still add stuff (locally)… ```pycon >>> s['something'] = 'else' >>> list(s) ['some', 'foo', 'hello', 'something'] ``` ### dol.caching.mk_write_cached_store(store=None, \*, w_cache=, flush_cache_condition=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Wrap a write cache around a store. * **Parameters:** * **w_cache** – The store to (write) cache to * **flush_cache_condition** – The condition to apply to the cache to decide whether it’s contents should be flushed or not A `w_cache` must have a clear method (that clears the cache’s contents). If you know what you’re doing and want to add one to your input kv store, you can do so by calling `ensure_clear_to_kv_store(store)` – this will add a `clear` method inplace AND return the resulting store as well. We didn’t add this automatically because the first thing `mk_write_cached_store` will do is call clear, to remove all the contents of the store. You don’t want to do this unwittingly and delete a bunch of precious data!! ```pycon >>> from dol.caching import mk_write_cached_store, ensure_clear_to_kv_store >>> from dol.base import Store >>> >>> def print_state(store): ... print(f"store: {store} ----- store._w_cache: {store._w_cache}") ... >>> class MyStore(dict): ... >>> MyCachedStore = mk_write_cached_store(MyStore, w_cache={}) # wrap MyStore with a (dict) write cache >>> s = MyCachedStore() # make a MyCachedStore instance >>> print_state(s) # print the contents (both store and cache), see that it's empty store: {} ----- store._w_cache: {} >>> s['hello'] = 'world' # write 'world' in 'hello' >>> print_state(s) # see that it hasn't been written store: {} ----- store._w_cache: {'hello': 'world'} >>> s['ding'] = 'dong' >>> print_state(s) store: {} ----- store._w_cache: {'hello': 'world', 'ding': 'dong'} >>> s.flush_cache() # manually flush the cache >>> print_state(s) # note that store._w_cache is empty, but store has the data now store: {'hello': 'world', 'ding': 'dong'} ----- store._w_cache: {} >>> >>> # But you usually want to use the store as a context manager >>> MyCachedStore = mk_write_cached_store( ... MyStore, w_cache={}, ... flush_cache_condition=None) >>> >>> the_persistent_dict = dict() >>> >>> s = MyCachedStore(the_persistent_dict) >>> with s: ... print("===> Before writing data:") ... print_state(s) ... s['hello'] = 'world' ... print("===> Before exiting the with block:") ... print_state(s) ... ===> Before writing data: store: {} ----- store._w_cache: {} ===> Before exiting the with block: store: {} ----- store._w_cache: {'hello': 'world'} >>> >>> print("===> After exiting the with block:"); print_state(s) # Note that the cache store flushed! ===> After exiting the with block: store: {'hello': 'world'} ----- store._w_cache: {} >>> >>> # Example of auto-flushing when there's at least two elements >>> class MyStore(dict): ... ... >>> MyCachedStore = mk_write_cached_store( ... MyStore, w_cache={}, ... flush_cache_condition=lambda w_cache: len(w_cache) >= 3) >>> >>> s = MyCachedStore() >>> with s: ... for i in range(7): ... s[i] = i * 10 ... print_state(s) ... store: {} ----- store._w_cache: {0: 0} store: {} ----- store._w_cache: {0: 0, 1: 10} store: {0: 0, 1: 10, 2: 20} ----- store._w_cache: {} store: {0: 0, 1: 10, 2: 20} ----- store._w_cache: {3: 30} store: {0: 0, 1: 10, 2: 20} ----- store._w_cache: {3: 30, 4: 40} store: {0: 0, 1: 10, 2: 20, 3: 30, 4: 40, 5: 50} ----- store._w_cache: {} store: {0: 0, 1: 10, 2: 20, 3: 30, 4: 40, 5: 50} ----- store._w_cache: {6: 60} >>> # There was still something left in the cache before exiting the with block. But now... >>> print_state(s) store: {0: 0, 1: 10, 2: 20, 3: 30, 4: 40, 5: 50, 6: 60} ----- store._w_cache: {} ``` ### dol.caching.register_key_strategy(cls) Register a class as a KeyStrategy. ### dol.caching.store_cached(store, key_func) Function output memorizer but using a specific (usually persisting) store as it’s memory and a key_func to compute the key under which to store the output. The key can be - a single value under which the output should be stored, regardless of the input. - a key function that is called on the inputs to create a hash under which the function’s output should be stored. * **Parameters:** * **store** – The key-value store to use for caching. Must support \_\_getitem_\_ and \_\_setitem_\_. * **key_func** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – The key function that is called on the input of the function to create the key value. #### SEE ALSO store_cached_with_single_key (for a version where the cache store key doesn’t depend on function’s args) ```pycon >>> # Note: Our doc test will use dict as the store, but to make the functionality useful beyond existing >>> # RAM-memorizer, you should use actual "persisting" stores that store in local files, or DBs, etc. >>> store = dict() >>> @store_cached(store, lambda *args: args) ... def my_data(x, y): ... print("Pretend this is a long computation") ... return x + y >>> t = my_data(1, 2) # note the print below (because the function is called Pretend this is a long computation >>> tt = my_data(1, 2) # note there's no print (because the function is NOT called) >>> assert t == tt >>> tt 3 >>> my_data(3, 4) # but different inputs will trigger the actual function again Pretend this is a long computation 7 >>> my_data._cache {(1, 2): 3, (3, 4): 7} ``` ### dol.caching.store_cached_with_single_key(store, key) Function output memorizer but using a specific store and key as its memory. Use in situations where you have a argument-less function or bound method that computes some data whose dependencies are static enough that there’s enough advantage to make the data refresh explicit (by deleting the cache entry) instead of making it implicit (recomputing/refetching the data every time). The key should be a single value under which the output should be stored, regardless of the input. #### NOTE The wrapped function comes with a empty_cache attribute, which when called, empties the cache (i.e. removes the key from the store) #### NOTE The wrapped function has a hidden `_cache` attribute pointing to the store in case you need to peep into it. * **Parameters:** * **store** – The cache. The key-value store to use for caching. Must support \_\_getitem_\_ and \_\_setitem_\_. * **key** – The store key under which to store the output of the function. #### SEE ALSO store_cached (for a version whose keys are computed from the wrapped function’s input. ```pycon >>> # Note: Our doc test will use dict as the store, but to make the functionality useful beyond existing >>> # RAM-memorizer, you should use actual "persisting" stores that store in local files, or DBs, etc. >>> store = dict() >>> @store_cached_with_single_key(store, 'whatevs') ... def my_data(): ... print("Pretend this is a long computation") ... return [1, 2, 3] >>> t = my_data() # note the print below (because the function is called Pretend this is a long computation >>> tt = my_data() # note there's no print (because the function is NOT called) >>> assert t == tt >>> tt [1, 2, 3] >>> my_data._cache # peep in the cache {'whatevs': [1, 2, 3]} >>> # let's empty the cache >>> my_data.empty_cache_entry() >>> assert 'whatevs' not in my_data._cache # see that the cache entry is gone. >>> t = my_data() # so when you call the function again, it prints again!d Pretend this is a long computation ``` # _autosummary/dol.content.html.md # dol.content Content references and content-addressed storage — the flat “blob” layer. Many apps split their data into two concerns (the *content-metadata bifurcation*; see `misc/docs/dol_content_metadata_bifurcation.md`): - **records / metadata** — small, queryable rows (a `MutableMapping` of dicts, a DB); - **content / blobs** — large bytes (media, documents, renders) that you don’t want to inline into a record or a query result. This module is the **content half**: a *flat* bytes store plus a small, serializable [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) token that stands in for the bytes inside a record. The store itself is just a `MutableMapping[str, bytes]` — so the backend is **injected** (`dict` in tests, `dol.Files` locally, an `s3dol` store in the cloud) and nothing here depends on any of them. Two addressing modes, mirroring the same convention used by the `zodal` TypeScript stores (so a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) serialized here matches `zodal`’s `ContentRef` on the wire — see [`ContentRef.to_json()`](_autosummary/dol.content.html.md#dol.content.ContentRef.to_json)): - **location-addressed** ([`put_content()`](_autosummary/dol.content.html.md#dol.content.put_content)) — the caller supplies the id; - **content-addressed** ([`add_content()`](_autosummary/dol.content.html.md#dol.content.add_content) / [`with_content_addressing()`](_autosummary/dol.content.html.md#dol.content.with_content_addressing)) — the id *is* the content hash, which makes writes idempotent and deduplicated (CAS). **URLs are resolved on demand, never baked in.** `put_content`/`add_content` leave `ContentRef.url` empty; call [`content_url()`](_autosummary/dol.content.html.md#dol.content.content_url) when you actually need a fetchable URL. This is deliberate: a backend’s `url_for` may mint a *presigned, expiring* URL (e.g. S3), and a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) is meant to be *persisted* inside a record — so freezing an expiring URL into it would be a latent bug. Reads can thus redirect to a CDN / presigned URL / static route while writes always go to the injected backend. ```pycon >>> store = {} >>> ref = add_content(store, b'hello world', name='greeting.txt') >>> ref.item_id == content_hash(b'hello world') True >>> (ref.size, ref.mime_type, ref.url) (11, 'text/plain', None) >>> get_content(store, ref) b'hello world' >>> is_content_ref(ref) and is_content_ref(ref.to_json()) True ``` The wire form is camelCase and drops empty fields, matching `zodal`’s `ContentRef`: ```pycon >>> ref.to_json()['itemId'] == ref.item_id True >>> sorted(ref.to_json()) ['_tag', 'field', 'hash', 'itemId', 'mimeType', 'size'] ``` ### Module Attributes | [`CONTENT_REF_TAG`](_autosummary/dol.content.html.md#dol.content.CONTENT_REF_TAG) | The `_tag` discriminator value carried on the JSON wire form (cross-language parity). | |--------------------------------------------------------------------|-----------------------------------------------------------------------------------------| | [`HashFunc`](_autosummary/dol.content.html.md#dol.content.HashFunc) | A key-minting hash constructor, e.g. `hashlib.sha256` — `bytes -> hash object`. | ### Functions | [`add_content`](_autosummary/dol.content.html.md#dol.content.add_content)(store, data, \*[, field, hasher, ...]) | Content-addressed write: the key *is* the content hash; idempotent (CAS). | |-------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| | [`content_hash`](_autosummary/dol.content.html.md#dol.content.content_hash)(data, \*[, hasher, length]) | Hex content hash of `data` (sha256 by default), optionally truncated to `length`. | | [`content_url`](_autosummary/dol.content.html.md#dol.content.content_url)(store, ref_or_key) | A fetchable URL for content, resolved **on demand**. | | [`delete_content`](_autosummary/dol.content.html.md#dol.content.delete_content)(store, ref_or_key) | Delete content by [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef), wire dict, or bare key (`del store[key]`). | | [`get_content`](_autosummary/dol.content.html.md#dol.content.get_content)(store, ref_or_key) | Read content bytes by [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef), wire dict, or bare key. | | [`guess_mime_type`](_autosummary/dol.content.html.md#dol.content.guess_mime_type)(name) | Guess a mime type from a filename/key by extension (stdlib `mimetypes`). | | [`is_content_ref`](_autosummary/dol.content.html.md#dol.content.is_content_ref)(obj) | True for a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) instance or its wire-form dict (`_tag` discriminator). | | [`put_content`](_autosummary/dol.content.html.md#dol.content.put_content)(store, item_id, data, \*[, ...]) | Location-addressed write: store `data` under a caller-supplied `item_id`. | | [`with_content_addressing`](_autosummary/dol.content.html.md#dol.content.with_content_addressing)([store, hasher, ...]) | Wrap an injected backend as a [`ContentAddressedStore`](_autosummary/dol.content.html.md#dol.content.ContentAddressedStore) (`dict` if `None`). | ### Classes | [`ContentAddressedStore`](_autosummary/dol.content.html.md#dol.content.ContentAddressedStore)([store, hasher, ...]) | A bytes store whose keys are the content hash of the values (CAS facade). | |------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------| | [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef)(item_id[, field, hash, url, ...]) | A small, serializable stand-in for stored content (bytes). | | [`SupportsUrlFor`](_autosummary/dol.content.html.md#dol.content.SupportsUrlFor)(\*args, \*\*kwargs) | A backend that can hand out a directly-fetchable URL for a stored key. | ### dol.content.CONTENT_REF_TAG *= 'ContentRef'* The `_tag` discriminator value carried on the JSON wire form (cross-language parity). ### *class* dol.content.ContentAddressedStore(store=None, \*, hasher=, length=None, field='content') Bases: [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister) A bytes store whose keys are the content hash of the values (CAS facade). Wraps any injected `MutableMapping` backend (`dict` for tests, `dol.Files` locally, an `s3dol` store in the cloud). Minting is via [`add()`](_autosummary/dol.content.html.md#dol.content.ContentAddressedStore.add) (the store picks the key); reads/iter/delete delegate to the backend. A direct `store[k] = v` is allowed only when `k` equals the content hash of `v` — so the CAS invariant can’t be silently violated. ```pycon >>> cas = with_content_addressing() # dict-backed >>> ref = cas.add(b'hello', name='h.txt') >>> cas[ref.item_id] b'hello' >>> list(cas) == [content_hash(b'hello')] True >>> cas.add(b'hello').item_id == ref.item_id # idempotent / deduplicated True ``` #### add(data, , mime_type=None, name=None) Write `data` under its content hash (idempotent); return a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef). * **Return type:** [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) #### *property* url_for Delegate the `url_for` seam to the backend if it has one (else `None`). ### *class* dol.content.ContentRef(item_id, field='content', hash=None, url=None, mime_type=None, size=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A small, serializable stand-in for stored content (bytes). Held *inside* a record in place of the bytes, so lists/queries stay light. It is addressed by `(item_id, field)` (a record may have several content fields); `hash` is populated for content-addressed writes and left `None` otherwise. `url` is an optional directly-fetchable location — normally left empty and resolved on demand via [`content_url()`](_autosummary/dol.content.html.md#dol.content.content_url) (see the module docstring). #### *classmethod* from_json(d) Parse a wire-form dict (camelCase) back into a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef). * **Return type:** [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) #### to_json() camelCase wire form matching `zodal`’s `ContentRef` (empty fields dropped). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### dol.content.HashFunc A key-minting hash constructor, e.g. `hashlib.sha256` — `bytes -> hash object`. alias of `Callable`[[[`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### *class* dol.content.SupportsUrlFor(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) A backend that can hand out a directly-fetchable URL for a stored key. The seam that lets **reads** redirect to a CDN / presigned URL / static route while **writes** stay on the injected backend. Local file stores typically don’t implement it ([`content_url()`](_autosummary/dol.content.html.md#dol.content.content_url) then returns `None`); an `s3dol` store implements it with a presigned URL — so all S3 knowledge lives in `s3dol`, never here. ### dol.content.add_content(store, data, \*, field='content', hasher=, length=None, mime_type=None, name=None) Content-addressed write: the key *is* the content hash; idempotent (CAS). A second call with identical bytes neither rewrites nor produces a different id, so identical content is stored once. Returns a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) with `hash` set. * **Return type:** [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) ```pycon >>> s = {} >>> a = add_content(s, b'xyz') >>> b = add_content(s, b'xyz') >>> a.item_id == b.item_id == a.hash and len(s) == 1 True ``` ### dol.content.content_hash(data, \*, hasher=, length=None) Hex content hash of `data` (sha256 by default), optionally truncated to `length`. Truncation trades key length for a higher collision probability (a 16-hex-char prefix is 64 bits) — leave `length` unset unless keys must be short and the corpus is small. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> content_hash(b'abc') == content_hash(b'abc') True >>> len(content_hash(b'abc', length=16)) 16 ``` ### dol.content.content_url(store, ref_or_key) A fetchable URL for content, resolved **on demand**. Prefers a URL the ref already carries; otherwise asks the backend’s `url_for` (the [`SupportsUrlFor`](_autosummary/dol.content.html.md#dol.content.SupportsUrlFor) seam), returning `None` if it has none. * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ```pycon >>> class Served(dict): ... def url_for(self, key): return f'https://cdn.example/{key}' >>> content_url(Served(), 'k1') 'https://cdn.example/k1' >>> content_url({}, 'k1') is None True >>> content_url({}, ContentRef('k1', url='https://carried/k1')) # ref carries its own 'https://carried/k1' ``` The key is resolved **through any wrapping layers**, so a URL addresses the same object `store[key]` reads. Without this, a key-transforming wrap would hand the backend the outer key and silently return a URL for a different object: ```pycon >>> from dol import KeyCodecs >>> wrapped = KeyCodecs.prefixed('a/')(Served)({'a/k1': b'v'}) >>> wrapped['k1'] b'v' >>> content_url(wrapped, 'k1') 'https://cdn.example/a/k1' ``` ### dol.content.delete_content(store, ref_or_key) Delete content by [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef), wire dict, or bare key (`del store[key]`). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ```pycon >>> s = {} >>> ref = add_content(s, b'gone') >>> delete_content(s, ref) >>> ref.item_id in s False ``` ### dol.content.get_content(store, ref_or_key) Read content bytes by [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef), wire dict, or bare key. * **Return type:** [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes) ```pycon >>> s = {} >>> ref = add_content(s, b'data') >>> get_content(s, ref) == get_content(s, ref.item_id) == b'data' True ``` ### dol.content.guess_mime_type(name) Guess a mime type from a filename/key by extension (stdlib `mimetypes`). Results depend on the platform’s mime registry, so treat them as best-effort. * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ```pycon >>> guess_mime_type('a.json') 'application/json' >>> guess_mime_type('no-extension') is None True ``` ### dol.content.is_content_ref(obj) True for a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) instance or its wire-form dict (`_tag` discriminator). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ```pycon >>> is_content_ref(ContentRef('id1')) True >>> is_content_ref({'_tag': 'ContentRef', 'itemId': 'id1'}) True >>> is_content_ref({'itemId': 'id1'}) or is_content_ref('id1') False ``` ### dol.content.put_content(store, item_id, data, , field='content', mime_type=None, name=None) Location-addressed write: store `data` under a caller-supplied `item_id`. Backend `store` is injected. Returns a [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) (mime guessed from `name` if not given; `url` left empty — resolve via [`content_url()`](_autosummary/dol.content.html.md#dol.content.content_url)). * **Return type:** [`ContentRef`](_autosummary/dol.content.html.md#dol.content.ContentRef) ```pycon >>> s = {} >>> ref = put_content(s, 'clip1', b'\x00\x01', name='clip1.wav') >>> ref.item_id, ref.hash, s['clip1'] ('clip1', None, b'\x00\x01') >>> ref.mime_type.startswith('audio/') True ``` ### dol.content.with_content_addressing(store=None, \*, hasher=, length=None, field='content') Wrap an injected backend as a [`ContentAddressedStore`](_autosummary/dol.content.html.md#dol.content.ContentAddressedStore) (`dict` if `None`). * **Return type:** [`ContentAddressedStore`](_autosummary/dol.content.html.md#dol.content.ContentAddressedStore) ```pycon >>> cas = with_content_addressing(length=16) >>> len(cas.add(b'abc').item_id) 16 ``` # _autosummary/dol.dig.html.md # dol.dig Layers introspection: walk the layers of a wrapped store and trace a key through them. Main entry points: - `layers`: the list of nested layers (found through the `store` attribute), outermost first - `trace_getitem`, `print_trace_info`: the (layer, method, value) steps of a `__getitem__` ### Functions | `dig_up`(store, attr[, default]) | | |--------------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `get_first_attr_found`(store, attrs[, default]) | | | [`inner_most`](_autosummary/dol.dig.html.md#dol.dig.inner_most)(store, arg, method[, default]) | The value of `arg` after every layer's `method` has been applied. | | `last_element`(gen) | | | `layers`(store[, layer_attrs]) | | | `next_layer`(store[, layer_attrs]) | | | `print_trace_info`(trace[, item_info]) | | | `print_trans_path`(store, arg, method[, with_type]) | | | `re_get_attr`(store, attr[, default]) | | | `recursive_calls`(func, x[, sentinel]) | | | `recursive_get_attr`(store, attr[, default]) | | | [`store_trans_path`](_autosummary/dol.dig.html.md#dol.dig.store_trans_path)(store, arg, method) | Yield `arg` transformed by `method` at each layer, outermost first. | | [`trace_getitem`](_autosummary/dol.dig.html.md#dol.dig.trace_getitem)(store, k[, layer_attrs]) | A generator of layered steps to inspect a store. | | `trace_info`(trace[, item_func]) | | ### dol.dig.inner_most(store, arg, method, default=) The value of `arg` after every layer’s `method` has been applied. ```pycon >>> from dol import wrap_kvs >>> s = wrap_kvs({'a.txt': 1}, id_of_key=lambda k: k + '.txt') >>> inner_most(s, 'a', '_id_of_key') 'a.txt' ``` \*\*Raises when no layer defines `method``**, instead of silently returning ``None`: ```pycon >>> inner_most({}, 'a', '_id_of_key') Traceback (most recent call last): ... AttributeError: No layer of dict defines '_id_of_key', so 'a' cannot be resolved. ... ``` A `None` here is the worst possible answer: callers use the result as a key or a path, so it surfaces far from its cause – as `https://.../None`, or as `TypeError: expected str, bytes or os.PathLike object, not NoneType`. Pass `default` to opt out of raising: ```pycon >>> inner_most({}, 'a', '_id_of_key', default=None) is None True ``` ### dol.dig.inner_most_key(store, arg, \*, method='_id_of_key', default=) The value of `arg` after every layer’s `method` has been applied. ```pycon >>> from dol import wrap_kvs >>> s = wrap_kvs({'a.txt': 1}, id_of_key=lambda k: k + '.txt') >>> inner_most(s, 'a', '_id_of_key') 'a.txt' ``` \*\*Raises when no layer defines `method``**, instead of silently returning ``None`: ```pycon >>> inner_most({}, 'a', '_id_of_key') Traceback (most recent call last): ... AttributeError: No layer of dict defines '_id_of_key', so 'a' cannot be resolved. ... ``` A `None` here is the worst possible answer: callers use the result as a key or a path, so it surfaces far from its cause – as `https://.../None`, or as `TypeError: expected str, bytes or os.PathLike object, not NoneType`. Pass `default` to opt out of raising: ```pycon >>> inner_most({}, 'a', '_id_of_key', default=None) is None True ``` ### dol.dig.inner_most_val(store, arg, \*, method='_data_of_obj', default=) The value of `arg` after every layer’s `method` has been applied. ```pycon >>> from dol import wrap_kvs >>> s = wrap_kvs({'a.txt': 1}, id_of_key=lambda k: k + '.txt') >>> inner_most(s, 'a', '_id_of_key') 'a.txt' ``` \*\*Raises when no layer defines `method``**, instead of silently returning ``None`: ```pycon >>> inner_most({}, 'a', '_id_of_key') Traceback (most recent call last): ... AttributeError: No layer of dict defines '_id_of_key', so 'a' cannot be resolved. ... ``` A `None` here is the worst possible answer: callers use the result as a key or a path, so it surfaces far from its cause – as `https://.../None`, or as `TypeError: expected str, bytes or os.PathLike object, not NoneType`. Pass `default` to opt out of raising: ```pycon >>> inner_most({}, 'a', '_id_of_key', default=None) is None True ``` ### dol.dig.store_trans_path(store, arg, method) Yield `arg` transformed by `method` at each layer, outermost first. Walks the `.store` chain, applying `store.` at every layer that defines it. ```pycon >>> from dol import wrap_kvs >>> s = wrap_kvs({'a.txt': 1}, id_of_key=lambda k: k + '.txt') >>> list(store_trans_path(s, 'a', '_id_of_key')) ['a.txt', 'a.txt'] ``` Yields nothing when no layer defines `method` – which is why [`inner_most()`](_autosummary/dol.dig.html.md#dol.dig.inner_most) raises rather than returning the `None` that an empty walk would otherwise produce. ```pycon >>> list(store_trans_path({}, 'a', '_id_of_key')) [] ``` ### dol.dig.trace_getitem(store, k, layer_attrs=('store',)) A generator of layered steps to inspect a store. * **Parameters:** * **store** – An instance that has the base.Store interface * **k** – A key * **layer_attrs** – The attribute names that should be checked to get the next layer. * **Returns:** A generator of (layer, method, value) We start with a small dict: ```pycon >>> d = {'a.num': '1000', 'b.num': '2000'} ``` Now let’s add layers to it. For example, with wrap_kvs: ```pycon >>> from dol.trans import wrap_kvs ``` Say that we want the interface to not see the `'.num'` strings, and deal with numerical values, not strings. ```pycon >>> s = wrap_kvs(d, ... key_of_id=lambda x: x[:-len('.num')], ... id_of_key=lambda x: x + '.num', ... obj_of_data=lambda x: int(x), ... data_of_obj=lambda x: str(x) ... ) >>> ``` Oh, and we want the interface to display upper case keys. ```pycon >>> ss = wrap_kvs(s, ... key_of_id=lambda x: x.upper(), ... id_of_key=lambda x: x.lower(), ... ) ``` And we want the numerical unit to be the kilo (that’s 1000): ```pycon >>> sss = wrap_kvs(ss, ... obj_of_data=lambda x: x / 1000, ... data_of_obj=lambda x: x * 1000 ... ) >>> >>> dict(sss.items()) {'A': 1.0, 'B': 2.0} ``` Well, if we had bugs, we’d like to inspect the various layers, and how they transform the data. ```python # Here's how to do that: # >>> for layer, method, value in trace_getitem(sss, 'A'): # ... print(layer, method, value) # ... # Traceback (most recent call last): # File "", line 1, in # NameError: name 'trace_getitem' is not defined ``` ```pycon >>> from dol.dig import trace_getitem >>> >>> for layer, method, value in trace_getitem(sss, 'A'): ... print(layer, method, value) ... {'a.num': '1000', 'b.num': '2000'} _id_of_key A {'a.num': '1000', 'b.num': '2000'} _id_of_key A {'a.num': '1000', 'b.num': '2000'} _id_of_key a {'a.num': '1000', 'b.num': '2000'} _id_of_key a {'a.num': '1000', 'b.num': '2000'} _id_of_key a.num {'a.num': '1000', 'b.num': '2000'} _id_of_key a.num {'a.num': '1000', 'b.num': '2000'} __getitem__ 1000 {'a.num': '1000', 'b.num': '2000'} _obj_of_data 1000 {'a.num': '1000', 'b.num': '2000'} _obj_of_data 1000 {'a.num': '1000', 'b.num': '2000'} _obj_of_data 1000 {'a.num': '1000', 'b.num': '2000'} _obj_of_data 1000 {'a.num': '1000', 'b.num': '2000'} _obj_of_data 1000 {'a.num': '1000', 'b.num': '2000'} _obj_of_data 1.0 ``` ### dol.dig.unravel_key(store, arg, , method='_id_of_key') Yield `arg` transformed by `method` at each layer, outermost first. Walks the `.store` chain, applying `store.` at every layer that defines it. ```pycon >>> from dol import wrap_kvs >>> s = wrap_kvs({'a.txt': 1}, id_of_key=lambda k: k + '.txt') >>> list(store_trans_path(s, 'a', '_id_of_key')) ['a.txt', 'a.txt'] ``` Yields nothing when no layer defines `method` – which is why [`inner_most()`](_autosummary/dol.dig.html.md#dol.dig.inner_most) raises rather than returning the `None` that an empty walk would otherwise produce. ```pycon >>> list(store_trans_path({}, 'a', '_id_of_key')) [] ``` ### dol.dig.unravel_val(store, arg, , method='_data_of_obj') Yield `arg` transformed by `method` at each layer, outermost first. Walks the `.store` chain, applying `store.` at every layer that defines it. ```pycon >>> from dol import wrap_kvs >>> s = wrap_kvs({'a.txt': 1}, id_of_key=lambda k: k + '.txt') >>> list(store_trans_path(s, 'a', '_id_of_key')) ['a.txt', 'a.txt'] ``` Yields nothing when no layer defines `method` – which is why [`inner_most()`](_autosummary/dol.dig.html.md#dol.dig.inner_most) raises rather than returning the `None` that an empty walk would otherwise produce. ```pycon >>> list(store_trans_path({}, 'a', '_id_of_key')) [] ``` # _autosummary/dol.errors.html.md # dol.errors Error objects and utils. The exception classes dol raises (`NotAllowed`, `OverWritesNotAllowedError`, `KeyValidationError`, …) and `items_with_caught_exceptions`, an `items()` that skips (or reports) the keys whose value cannot be fetched. ```pycon >>> from dol.errors import items_with_caught_exceptions >>> list(items_with_caught_exceptions({'a': 1})) [('a', 1)] ``` ### Functions | [`items_with_caught_exceptions`](_autosummary/dol.errors.html.md#dol.errors.items_with_caught_exceptions)(d[, callback, ...]) | Do what Mapping.items() does, but catching exceptions when getting the values for a key. | |-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------| ### Exceptions | [`AlreadyExists`](_autosummary/dol.errors.html.md#dol.errors.AlreadyExists) | To use if an object already exists (and shouldn't; for example, to protect overwrites) | |----------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------| | [`DeletionsNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.DeletionsNotAllowed) | Delete OperationNotAllowed | | [`IterationNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.IterationNotAllowed) | Iteration OperationNotAllowed | | [`KeyValidationError`](_autosummary/dol.errors.html.md#dol.errors.KeyValidationError) | Error to raise when a key is not valid | | [`MethodFuncNotValid`](_autosummary/dol.errors.html.md#dol.errors.MethodFuncNotValid) | Use when method function is not valid | | [`MethodNameAlreadyExists`](_autosummary/dol.errors.html.md#dol.errors.MethodNameAlreadyExists) | To use when a method name already exists (and shouldn't) | | [`NoSuchKeyError`](_autosummary/dol.errors.html.md#dol.errors.NoSuchKeyError) | When a requested key doesn't exist | | [`NotAllowed`](_autosummary/dol.errors.html.md#dol.errors.NotAllowed) | To use to indicate that something is not allowed | | [`NotValid`](_autosummary/dol.errors.html.md#dol.errors.NotValid) | To use to indicate when an object doesn't fit expected properties | | [`OperationNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.OperationNotAllowed) | When a given operation is not allowed (through being disabled, conditioned, or just implemented) | | [`OverWritesNotAllowedError`](_autosummary/dol.errors.html.md#dol.errors.OverWritesNotAllowedError) | Error to raise when a writes to existing keys are not allowed | | [`ReadsNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.ReadsNotAllowed) | Read OperationNotAllowed | | [`SetattrNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.SetattrNotAllowed) | An attribute was requested to be set, but some conditions didn't apply | | [`WritesNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.WritesNotAllowed) | Write OperationNotAllowed | ### *exception* dol.errors.AlreadyExists Bases: [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) To use if an object already exists (and shouldn’t; for example, to protect overwrites) ### *exception* dol.errors.DeletionsNotAllowed Bases: [`OperationNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.OperationNotAllowed) Delete OperationNotAllowed ### *exception* dol.errors.IterationNotAllowed Bases: [`OperationNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.OperationNotAllowed) Iteration OperationNotAllowed ### *exception* dol.errors.KeyValidationError Bases: [`NotValid`](_autosummary/dol.errors.html.md#dol.errors.NotValid) Error to raise when a key is not valid ### *exception* dol.errors.MethodFuncNotValid Bases: [`NotValid`](_autosummary/dol.errors.html.md#dol.errors.NotValid) Use when method function is not valid ### *exception* dol.errors.MethodNameAlreadyExists Bases: [`AlreadyExists`](_autosummary/dol.errors.html.md#dol.errors.AlreadyExists) To use when a method name already exists (and shouldn’t) ### *exception* dol.errors.NoSuchKeyError Bases: [`KeyError`](https://docs.python.org/3/builtins/exceptions.html#KeyError) When a requested key doesn’t exist ### *exception* dol.errors.NotAllowed Bases: [`Exception`](https://docs.python.org/3/builtins/exceptions.html#Exception) To use to indicate that something is not allowed ### *exception* dol.errors.NotValid Bases: [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError), [`TypeError`](https://docs.python.org/3/builtins/exceptions.html#TypeError) To use to indicate when an object doesn’t fit expected properties ### *exception* dol.errors.OperationNotAllowed Bases: [`NotAllowed`](_autosummary/dol.errors.html.md#dol.errors.NotAllowed), [`NotImplementedError`](https://docs.python.org/3/builtins/exceptions.html#NotImplementedError) When a given operation is not allowed (through being disabled, conditioned, or just implemented) ### *exception* dol.errors.OverWritesNotAllowedError Bases: [`OperationNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.OperationNotAllowed) Error to raise when a writes to existing keys are not allowed ### *exception* dol.errors.ReadsNotAllowed Bases: [`OperationNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.OperationNotAllowed) Read OperationNotAllowed ### *exception* dol.errors.SetattrNotAllowed Bases: [`NotAllowed`](_autosummary/dol.errors.html.md#dol.errors.NotAllowed) An attribute was requested to be set, but some conditions didn’t apply ### *exception* dol.errors.WritesNotAllowed Bases: [`OperationNotAllowed`](_autosummary/dol.errors.html.md#dol.errors.OperationNotAllowed) Write OperationNotAllowed ### dol.errors.items_with_caught_exceptions(d, callback=None, catch_exceptions=(, ), yield_callback_output=False) Do what Mapping.items() does, but catching exceptions when getting the values for a key. Some time your `store.items()` is annoying because of some exceptions that happen when you’re retrieving some value for some of the keys. Yes, if that happens, it’s that something is wrong with your store, and yes, if it’s a store that’s going to be used a lot, you really should build the right store that doesn’t have that problem. But now that we appeased the paranoid naysayers with that warning, let’s get to business: Sometimes you just want to get through the hurdle to get the job done. Sometimes your store is good enough, except for a few exceptions. Sometimes your store gets it’s keys from a large pool of possible keys (e.g. github stores or kaggle stores, or any store created by a free-form search seed), so you can’t really depend on the fact that all the keys given by your key iterator will give you a value without exception – especially if you slapped on a bunch of post-processing on the out-coming values. So you can right a for loop to iterate over your keys, catch the exceptions, do something with it… Or, in many cases, you can just use `items_with_caught_exceptions`. * **Parameters:** * **d** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – Any Mapping * **catch_exceptions** – A tuple of exceptions that should be caught * **callback** – A function that will be called every time an exception is caught. It may take any subset of the arguments `k` (key), `e` (error obj), `d` (mapping) and `i` (index), by name (see the examples below); if its signature cannot be inspected it is called with all four, positionally. * **yield_callback_output** – If True, also yield the callback’s output for the keys whose value raised. * **Returns:** An (key, val) generator with exceptions caught ```pycon >>> from collections.abc import Mapping >>> class Test(Mapping): # a Mapping class that has keys 0..9, but raises of KeyError if the key is not even ... n = 10 ... def __iter__(self): ... yield from range(2, self.n) ... def __len__(self): ... return self.n ... def __getitem__(self, k): ... if k % 2 == 0: ... return k ... else: ... raise KeyError('Not even') >>> >>> list(items_with_caught_exceptions(Test())) [(2, 2), (4, 4), (6, 6), (8, 8)] >>> >>> def my_log(k, e): ... print(k, e) >>> list(items_with_caught_exceptions(Test(), callback=my_log)) 3 'Not even' 5 'Not even' 7 'Not even' 9 'Not even' [(2, 2), (4, 4), (6, 6), (8, 8)] >>> def my_other_log(i): ... print(i) >>> list(items_with_caught_exceptions(Test(), callback=my_other_log)) 1 3 5 7 [(2, 2), (4, 4), (6, 6), (8, 8)] ``` # _autosummary/dol.explicit.html.md # dol.explicit Stores whose keys are given explicitly, with values fetched lazily from a source. Main entry points: - `KeysReader`: a collection of keys plus a `getter(src, key)` - `ExplicitKeysSource`: explicit keys plus a function reading the value for a key - `ExplicitKeysStore`: wrap a store so that its keys come from an explicit iterable - `ExplicitKeyMap`: a key mapper given as explicit dicts ```pycon >>> from dol.explicit import KeysReader >>> r = KeysReader({'apple': 'pie', 'banana': 'split'}, ['banana'], lambda src, k: src[k]) >>> list(r), r['banana'] (['banana'], 'split') ``` ### Classes | [`ExplicitKeyMap`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeyMap)(\*[, key_of_id, id_of_key]) | A key mapper given as explicit `key_of_id`/`id_of_key` dicts (one is enough; the other is derived, and both are checked to be inverse of each other). | |-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| | [`ExplicitKeymapReader`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeymapReader)(store[, key_of_id, ...]) | Wrap a store (instance) so that it gets it's keys from an explicit iterable of keys. | | [`ExplicitKeys`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeys)(key_collection) | dol.base.Keys implementation that gets it's keys explicitly from a collection given at initialization time. | | [`ExplicitKeysSource`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeysSource)(key_collection, \_obj_of_key) | An object source that uses an explicit keys collection and a specified function to read contents for a key. | | [`ExplicitKeysStore`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeysStore)(store, key_collection) | Wrap a store (instance) so that it gets it's keys from an explicit iterable of keys. | | [`KeysReader`](_autosummary/dol.explicit.html.md#dol.explicit.KeysReader)(src, key_collection, getter, \*[, ...]) | Mapping defined by keys with a getter function that gets values from keys. | | `ObjDumper`(save_data_to_key[, data_of_obj]) | | ### *class* dol.explicit.ExplicitKeyMap(, key_of_id=None, id_of_key=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A key mapper given as explicit `key_of_id`/`id_of_key` dicts (one is enough; the other is derived, and both are checked to be inverse of each other). Provides the `_key_of_id`/`_id_of_key` methods that `kv_wrap` looks for. ### *class* dol.explicit.ExplicitKeymapReader(store, key_of_id=None, id_of_key=None) Bases: [`ExplicitKeys`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeys), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) Wrap a store (instance) so that it gets it’s keys from an explicit iterable of keys. ```pycon >>> s = {'a': 1, 'b': 2, 'c': 3, 'd': 4} >>> id_of_key = {'A': 'a', 'C': 'c'} >>> ss = ExplicitKeymapReader(s, id_of_key=id_of_key) >>> list(ss) ['A', 'C'] >>> ss['C'] # will look up 'C', find 'c', and call the store on that. 3 ``` ### *class* dol.explicit.ExplicitKeys(key_collection) Bases: [`Collection`](_autosummary/dol.base.html.md#dol.base.Collection) dol.base.Keys implementation that gets it’s keys explicitly from a collection given at initialization time. The key_collection must be a collections.abc.Collection (such as list, tuple, set, etc.) ```pycon >>> keys = ExplicitKeys(key_collection=['foo', 'bar', 'alice']) >>> 'foo' in keys True >>> 'not there' in keys False >>> list(keys) ['foo', 'bar', 'alice'] ``` ### *class* dol.explicit.ExplicitKeysSource(key_collection, \_obj_of_key) Bases: [`ExplicitKeys`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeys), [`ObjReader`](_autosummary/dol.sources.html.md#dol.sources.ObjReader), [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) An object source that uses an explicit keys collection and a specified function to read contents for a key. ```pycon >>> s = ExplicitKeysSource([1, 2, 3], str) >>> list(s) [1, 2, 3] >>> list(s.values()) ['1', '2', '3'] ``` Main functionality equivalent to recipe: ```pycon >>> def explicit_keys_source(key_collection, _obj_of_key): ... from dol.trans import wrap_kvs ... return wrap_kvs({k: k for k in key_collection}, obj_of_data=_obj_of_key) ``` ```pycon >>> s = explicit_keys_source([1, 2, 3], str) >>> list(s) [1, 2, 3] >>> list(s.values()) ['1', '2', '3'] ``` ### *class* dol.explicit.ExplicitKeysStore(store, key_collection) Bases: [`ExplicitKeys`](_autosummary/dol.explicit.html.md#dol.explicit.ExplicitKeys), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) Wrap a store (instance) so that it gets it’s keys from an explicit iterable of keys. ```pycon >>> s = {'a': 1, 'b': 2, 'c': 3, 'd': 4} >>> list(s) ['a', 'b', 'c', 'd'] >>> ss = ExplicitKeysStore(s, ['d', 'a']) >>> len(ss) 2 >>> list(ss) ['d', 'a'] >>> list(ss.values()) [4, 1] >>> ss.head() ('d', 4) ``` ### *class* dol.explicit.KeysReader(src, key_collection, getter, \*, key_error_msg=) Bases: [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) Mapping defined by keys with a getter function that gets values from keys. `KeysReader` is particularly useful in cases where you want to have a mapping that lazy-load values for keys from an explicit collection. Keywords: Lazy-evaluation, Mapping * **Parameters:** * **src** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Source`)) – The source where values will be extracted from. * **key_collection** ([`Collection`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)]) – A collection of keys that will be used to extract values from `src`. * **getter** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Source`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)]) – A function that takes a source and a key, and returns the value for that key. * **key_error_msg** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Source`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A function that takes a source and a key, and returns an error message. ### Example ```pycon >>> src = {'apple': 'pie', 'banana': 'split', 'carrot': 'cake'} >>> key_collection = ['carrot', 'apple'] >>> getter = lambda src, key: src[key] >>> key_reader = KeysReader(src, key_collection, getter) ``` Note that the only the keys mentioned by `key_collection` will be iterated through, and in the order they are mentioned in `key_collection`. ```pycon >>> list(key_reader) ['carrot', 'apple'] ``` ```pycon >>> key_reader['apple'] 'pie' >>> key_reader['banana'] Traceback (most recent call last): ... KeyError: "Key 'banana' was not found....key_collection attribute)" ``` Let’s take the same `src` and `key_collection`, but with a different getter and key_error_msg: Note that a key_error_msg must be a function that takes a `src` and a `key`, in that order and with those argument names. Say you wanted to not use the `src` in your message. You would still have to write a function that takes `src` as the first argument. ```pycon >>> key_error_msg = lambda src, key: f"Key {key} was not found" # no source information ``` ```pycon >>> getter = lambda src, key: f"Value for {key} in {src}: {src[key]}" >>> key_reader = KeysReader(src, key_collection, getter, key_error_msg=key_error_msg) >>> list(key_reader) ['carrot', 'apple'] >>> key_reader['apple'] "Value for apple in {'apple': 'pie', 'banana': 'split', 'carrot': 'cake'}: pie" >>> key_reader['banana'] Traceback (most recent call last): ... KeyError: "Key banana was not found" ``` # _autosummary/dol.filesys.html.md # dol.filesys File system access: dict-like stores over folders and files. `Files` gives a folder a `MutableMapping` interface: keys are paths relative to the root folder, values are the files’ bytes. `TextFiles`, `JsonFiles` and `PickleFiles` add the corresponding value codecs. Writing under a sub-folder that does not exist raises `KeyError`; wrap the store with `mk_dirs_if_missing` to create folders on write. Main entry points: - `Files`: bytes of the files under a root folder - `TextFiles`: same, with text values - `JsonFiles`: same, with JSON-decoded values - `PickleFiles`: same, with pickled values - `mk_dirs_if_missing`: make a file store create missing directories on write ```pycon >>> import tempfile >>> s = Files(tempfile.mkdtemp()) >>> s['hello.txt'] = b'world' >>> s['hello.txt'] b'world' >>> list(s) ['hello.txt'] ``` ### Functions | [`create_directories`](_autosummary/dol.filesys.html.md#dol.filesys.create_directories)(dirpath[, max_dirs_to_make]) | Create directories up to a specified limit. | |------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`ensure_dir`](_autosummary/dol.filesys.html.md#dol.filesys.ensure_dir)(dirpath, \*[, max_dirs_to_make, ...]) | Ensure that a directory exists, creating it if necessary. | | [`ensure_slash_suffix`](_autosummary/dol.filesys.html.md#dol.filesys.ensure_slash_suffix)(path) | Add a file separation (/ or ) at the end of path str, if not already present. | | [`iter_dirpaths_in_folder_recursively`](_autosummary/dol.filesys.html.md#dol.filesys.iter_dirpaths_in_folder_recursively)(root_folder) | Recursively generates dirpaths of folder (and subfolders, etc.) up to a given level | | [`iter_filepaths_in_folder_recursively`](_autosummary/dol.filesys.html.md#dol.filesys.iter_filepaths_in_folder_recursively)(root_folder) | Recursively generates filepaths of folder (and subfolders, etc.) up to a given level | | [`mk_absolute_path`](_autosummary/dol.filesys.html.md#dol.filesys.mk_absolute_path)(path_format) | Expand a leading `~`, or make a leading `.` path absolute; other paths are returned as is. | | [`mk_dirs_if_missing`](_autosummary/dol.filesys.html.md#dol.filesys.mk_dirs_if_missing)([store_cls, ...]) | Store decorator that will make the store create directories on write as needed. | | [`mk_dirs_if_missing_preset`](_autosummary/dol.filesys.html.md#dol.filesys.mk_dirs_if_missing_preset)(self, k, v, \*[, ...]) | Preset function that will make the store create directories on write as needed. | | [`mk_json_bytes_wrap`](_autosummary/dol.filesys.html.md#dol.filesys.mk_json_bytes_wrap)(\*[, loads_kwargs, ...]) | Make a `wrap_kvs` value-codec wrapper for JSON, with kwargs for `json.loads`/`json.dumps`. | | [`mk_pickle_bytes_wrap`](_autosummary/dol.filesys.html.md#dol.filesys.mk_pickle_bytes_wrap)(\*[, loads_kwargs, ...]) | Make a `wrap_kvs` value-codec wrapper for pickle, with kwargs for `pickle.loads`/`pickle.dumps`. | | [`mk_tmp_dol_dir`](_autosummary/dol.filesys.html.md#dol.filesys.mk_tmp_dol_dir)([dirname, ...]) | Create and return a path to a temporary directory that's guaranteed to be accessible to the user. | | [`paths_in_dir`](_autosummary/dol.filesys.html.md#dol.filesys.paths_in_dir)(rootdir[, include_hidden]) | Yield the paths of the entries of `rootdir` (directories with a trailing separator), skipping hidden ones unless `include_hidden`. | | [`process_path`](_autosummary/dol.filesys.html.md#dol.filesys.process_path)(\*path[, ensure_dir_exists, ...]) | Process a path string, ensuring it exists, and optionally expanding user. | | [`resolve_dir`](_autosummary/dol.filesys.html.md#dol.filesys.resolve_dir)(dirpath[, assert_existence, ...]) | Resolve a path to a full, real, path to a directory | | [`resolve_path`](_autosummary/dol.filesys.html.md#dol.filesys.resolve_path)(path[, assert_existence]) | Resolve a path to a full, real, (file or folder) path (opt assert existence). | | [`subfolder_stores`](_autosummary/dol.filesys.html.md#dol.filesys.subfolder_stores)(root_folder, \*[, ...]) | Create a store of subfolders of a given folder, where the keys are the subfolder paths (by default, relative and slash-less) and the values are stores of these subfolders. | | [`temp_dir`](_autosummary/dol.filesys.html.md#dol.filesys.temp_dir)([dirname, make_it_if_necessary, ...]) | Create and return a path to a temporary directory that's guaranteed to be accessible to the user. | | [`validate_key_and_raise_key_error_on_exception`](_autosummary/dol.filesys.html.md#dol.filesys.validate_key_and_raise_key_error_on_exception)(func) | Method decorator: validate the key first, and re-raise any exception of the method as a `KeyError`. | ### Classes | [`DirCollection`](_autosummary/dol.filesys.html.md#dol.filesys.DirCollection)(rootdir[, subpath, ...]) | Collection of the directory paths under `rootdir`. | |---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | [`DirReader`](_autosummary/dol.filesys.html.md#dol.filesys.DirReader)(rootdir[, subpath, ...]) | Reader mapping each sub-directory of `rootdir` to a `DirReader` of it. | | [`FileBytesPersister`](_autosummary/dol.filesys.html.md#dol.filesys.FileBytesPersister)(\*args[, delete_func]) | File persistence with configurable deletion. | | [`FileBytesReader`](_autosummary/dol.filesys.html.md#dol.filesys.FileBytesReader)(rootdir[, subpath, ...]) | Reader mapping file paths under `rootdir` to the files' bytes. | | [`FileCollection`](_autosummary/dol.filesys.html.md#dol.filesys.FileCollection)(rootdir[, subpath, ...]) | Collection of the file paths under `rootdir`. | | [`FileInfoReader`](_autosummary/dol.filesys.html.md#dol.filesys.FileInfoReader)(rootdir[, subpath, ...]) | Reader mapping file paths to their `os.stat` result. | | [`FileStringPersister`](_autosummary/dol.filesys.html.md#dol.filesys.FileStringPersister)(\*args[, delete_func]) | Persister mapping file paths to the files' text (files opened in text mode). | | [`FileStringReader`](_autosummary/dol.filesys.html.md#dol.filesys.FileStringReader)(rootdir[, subpath, ...]) | Reader mapping file paths to the files' text (files opened in text mode). | | [`FileSysCollection`](_autosummary/dol.filesys.html.md#dol.filesys.FileSysCollection)(rootdir[, subpath, ...]) | Base collection of file-system paths under `rootdir`, optionally restricted by `subpath`, `max_levels` and hidden-file inclusion. | | [`Files`](_autosummary/dol.filesys.html.md#dol.filesys.Files)(\*args[, delete_func]) | FileBytesPersister with relative paths | | [`FilesReader`](_autosummary/dol.filesys.html.md#dol.filesys.FilesReader)(rootdir[, subpath, ...]) | FileBytesReader with relative paths | | [`JsonFiles`](_autosummary/dol.filesys.html.md#dol.filesys.JsonFiles)(\*args[, delete_func]) | A store of json files | | [`Jsons`](_autosummary/dol.filesys.html.md#dol.filesys.Jsons)(\*args[, delete_func]) | Like JsonFiles, but with added .json extension handling Namely: filtering for `.json` extensions but not showing the extension in keys | | [`LocalFileDeleteMixin`](_autosummary/dol.filesys.html.md#dol.filesys.LocalFileDeleteMixin)() | Mixin providing configurable file deletion. | | [`MakeMissingDirsStoreMixin`](_autosummary/dol.filesys.html.md#dol.filesys.MakeMissingDirsStoreMixin)() | Will make a local file store automatically create the directories needed to create a file. | | [`PickleFiles`](_autosummary/dol.filesys.html.md#dol.filesys.PickleFiles)(\*args[, delete_func]) | A store of pickles | | [`PickleStore`](_autosummary/dol.filesys.html.md#dol.filesys.PickleStore) | | | [`PickleStores`](_autosummary/dol.filesys.html.md#dol.filesys.PickleStores)(rootdir[, subpath, ...]) | Reader mapping each sub-directory of `rootdir` to a `PickleFiles` store of it. | | [`RelPathFileBytesPersister`](_autosummary/dol.filesys.html.md#dol.filesys.RelPathFileBytesPersister) | | | [`RelPathFileBytesReader`](_autosummary/dol.filesys.html.md#dol.filesys.RelPathFileBytesReader) | | | [`RelPathFileStringPersister`](_autosummary/dol.filesys.html.md#dol.filesys.RelPathFileStringPersister) | | | [`RelPathFileStringReader`](_autosummary/dol.filesys.html.md#dol.filesys.RelPathFileStringReader) | | | [`ReprMixin`](_autosummary/dol.filesys.html.md#dol.filesys.ReprMixin)() | A `__repr__` showing the `_init_kwargs` the instance was created with. | | [`TextFiles`](_autosummary/dol.filesys.html.md#dol.filesys.TextFiles)(\*args[, delete_func]) | FileStringPersister with relative paths | | [`TextFilesReader`](_autosummary/dol.filesys.html.md#dol.filesys.TextFilesReader)(rootdir[, subpath, ...]) | FileStringReader with relative paths | ### Exceptions | [`KeyValidationError`](_autosummary/dol.filesys.html.md#dol.filesys.KeyValidationError) | A `KeyError` for keys that fail a file-system store's validation. | |-----------------------------------------------------------------------|---------------------------------------------------------------------| ### *class* dol.filesys.DirCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`FileSysCollection`](_autosummary/dol.filesys.html.md#dol.filesys.FileSysCollection) Collection of the directory paths under `rootdir`. ### *class* dol.filesys.DirReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`DirCollection`](_autosummary/dol.filesys.html.md#dol.filesys.DirCollection), [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) Reader mapping each sub-directory of `rootdir` to a `DirReader` of it. ### *class* dol.filesys.FileBytesPersister(\*args, delete_func=None, \*\*kwargs) Bases: [`LocalFileDeleteMixin`](_autosummary/dol.filesys.html.md#dol.filesys.LocalFileDeleteMixin), [`FileBytesReader`](_autosummary/dol.filesys.html.md#dol.filesys.FileBytesReader), [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister) File persistence with configurable deletion. Supports custom deletion functions via delete_func parameter in \_\_init_\_. By default, tries to move files to trash with fallback to os.remove. See dol.trash module for deletion strategies: permanent_delete, trash_only, etc. ### *class* dol.filesys.FileBytesReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`FileCollection`](_autosummary/dol.filesys.html.md#dol.filesys.FileCollection), [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) Reader mapping file paths under `rootdir` to the files’ bytes. ### *class* dol.filesys.FileCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`FileSysCollection`](_autosummary/dol.filesys.html.md#dol.filesys.FileSysCollection) Collection of the file paths under `rootdir`. ### *class* dol.filesys.FileInfoReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`FileCollection`](_autosummary/dol.filesys.html.md#dol.filesys.FileCollection), [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) Reader mapping file paths to their `os.stat` result. ### *class* dol.filesys.FileStringPersister(\*args, delete_func=None, \*\*kwargs) Bases: [`FileBytesPersister`](_autosummary/dol.filesys.html.md#dol.filesys.FileBytesPersister) Persister mapping file paths to the files’ text (files opened in text mode). Reads and writes as UTF-8 explicitly, rather than inheriting `locale.getpreferredencoding()` (see i2mint/dol#97): a store is a serialization boundary, and one whose format silently depends on an ambient environment variable isn’t really specified. Without this, a write can raise on a non-ASCII-locale machine, or a store synced between two machines with different locales can silently corrupt on round trip. ### *class* dol.filesys.FileStringReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`FileBytesReader`](_autosummary/dol.filesys.html.md#dol.filesys.FileBytesReader) Reader mapping file paths to the files’ text (files opened in text mode). Reads as UTF-8 explicitly, rather than inheriting `locale.getpreferredencoding()` (see i2mint/dol#97): a store is a serialization boundary, and one whose format silently depends on an ambient environment variable isn’t really specified. This also matches how `FileStringPersister` writes (below), so a round trip is safe regardless of which locale reads or writes. ### *class* dol.filesys.FileSysCollection(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`Collection`](_autosummary/dol.base.html.md#dol.base.Collection) Base collection of file-system paths under `rootdir`, optionally restricted by `subpath`, `max_levels` and hidden-file inclusion. #### with_relative_paths() Return a copy of self with relative paths ### *class* dol.filesys.Files(\*args, delete_func=None, \*\*kwargs) Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) FileBytesPersister with relative paths #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### *class* dol.filesys.FilesReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) FileBytesReader with relative paths #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### *class* dol.filesys.JsonFiles(\*args, delete_func=None, \*\*kwargs) Bases: [`Store`](_autosummary/dol.base.html.md#dol.base.Store) A store of json files #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### *class* dol.filesys.Jsons(\*args, delete_func=None, \*\*kwargs) Bases: [`Store`](_autosummary/dol.base.html.md#dol.base.Store) Like JsonFiles, but with added .json extension handling Namely: filtering for `.json` extensions but not showing the extension in keys #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### *exception* dol.filesys.KeyValidationError Bases: [`KeyError`](https://docs.python.org/3/builtins/exceptions.html#KeyError) A `KeyError` for keys that fail a file-system store’s validation. ### *class* dol.filesys.LocalFileDeleteMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mixin providing configurable file deletion. The deletion function can be configured either at class level by setting the \_delete_func class attribute, or at instance level by setting the \_delete_func instance attribute. By default, uses safe deletion that tries to move to trash with fallback to os.remove (with warning). See dol.trash module for available deletion strategies: - default_delete_func: Safe trash with warning on fallback - permanent_delete: Direct os.remove (no warnings) - trash_only: Error if trash unavailable ### *class* dol.filesys.MakeMissingDirsStoreMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Will make a local file store automatically create the directories needed to create a file. Should be placed before the concrete perisister in the mro but in such a manner so that it receives full paths. ### *class* dol.filesys.PickleFiles(\*args, delete_func=None, \*\*kwargs) Bases: [`Store`](_autosummary/dol.base.html.md#dol.base.Store) A store of pickles #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### dol.filesys.PickleStore alias of [`PickleFiles`](_autosummary/dol.filesys.html.md#dol.filesys.PickleFiles) ### *class* dol.filesys.PickleStores(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) Reader mapping each sub-directory of `rootdir` to a `PickleFiles` store of it. #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### dol.filesys.RelPathFileBytesPersister alias of [`Files`](_autosummary/dol.filesys.html.md#dol.filesys.Files) ### dol.filesys.RelPathFileBytesReader alias of [`FilesReader`](_autosummary/dol.filesys.html.md#dol.filesys.FilesReader) ### dol.filesys.RelPathFileStringPersister alias of [`TextFiles`](_autosummary/dol.filesys.html.md#dol.filesys.TextFiles) ### dol.filesys.RelPathFileStringReader alias of [`TextFilesReader`](_autosummary/dol.filesys.html.md#dol.filesys.TextFilesReader) ### *class* dol.filesys.ReprMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A `__repr__` showing the `_init_kwargs` the instance was created with. ### *class* dol.filesys.TextFiles(\*args, delete_func=None, \*\*kwargs) Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) FileStringPersister with relative paths #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### *class* dol.filesys.TextFilesReader(rootdir, subpath='', pattern_for_field=None, max_levels=None, , include_hidden=False, assert_rootdir_existence=False) Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) FileStringReader with relative paths #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### dol.filesys.create_directories(dirpath, max_dirs_to_make=None) Create directories up to a specified limit. * **Parameters:** * **dirpath** – The directory path to create. * **max_dirs_to_make** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The maximum number of directories to create. If None, there’s no limit. * **Returns:** True if the directory exists (already, or after creation); False if creating it would need more than `max_dirs_to_make` new directories (none are made). * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If max_dirs_to_make is negative. ### Examples ```pycon >>> import tempfile, shutil >>> temp_dir = tempfile.mkdtemp() >>> target_dir = os.path.join(temp_dir, 'a', 'b', 'c') >>> create_directories(target_dir, max_dirs_to_make=2) False >>> create_directories(target_dir, max_dirs_to_make=3) True >>> os.path.isdir(target_dir) True >>> shutil.rmtree(temp_dir) # Cleanup ``` ```pycon >>> temp_dir = tempfile.mkdtemp() >>> target_dir = os.path.join(temp_dir, 'a', 'b', 'c', 'd') >>> create_directories(target_dir) True >>> os.path.isdir(target_dir) True >>> shutil.rmtree(temp_dir) # Cleanup ``` ### dol.filesys.ensure_dir(dirpath, , max_dirs_to_make=None, verbose=False) Ensure that a directory exists, creating it if necessary. * **Parameters:** * **dirpath** – path to the directory to create * **max_dirs_to_make** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the maximum number of directories to create. If None, there’s no limit. * **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool) | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – controls verbosity (the noise ensure_dir makes if it make folder) * **Returns:** the path to the directory When the path does not exist, if `verbose` is: - a `bool`’ a standard message will be printed - a `callable`; will be called on dirpath before directory is created – you can use this to ask the user for confirmation for example - a ‘’string\`\`; this string will be printed Usage note: If you want to string or the (argument-less) callable to be dependent on `dirpath`, you need make them so when calling ensure_dir. ### dol.filesys.ensure_slash_suffix(path) Add a file separation (/ or ) at the end of path str, if not already present. An empty path stays empty: an empty prefix has no slash to “ensure”, and turning it into a bare separator anchors otherwise-absolute keys to the filesystem root. On Windows that produces invalid paths like `\C:\Users\...` (a separator before the drive letter -> `OSError: [Errno 22]`); e.g. `Files("")` used with absolute keys, as in `dol.misc.get_obj`. ### dol.filesys.iter_dirpaths_in_folder_recursively(root_folder, max_levels=None, \_current_level=0, include_hidden=False) Recursively generates dirpaths of folder (and subfolders, etc.) up to a given level ### dol.filesys.iter_filepaths_in_folder_recursively(root_folder, max_levels=None, \_current_level=0, include_hidden=False) Recursively generates filepaths of folder (and subfolders, etc.) up to a given level ### dol.filesys.mk_absolute_path(path_format) Expand a leading `~`, or make a leading `.` path absolute; other paths are returned as is. ### dol.filesys.mk_dirs_if_missing(store_cls=None, , max_dirs_to_make=None, verbose=False, key_condition=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Store decorator that will make the store create directories on write as needed. Note that it’ll only effect paths relative to the rootdir, which needs to be ensured to exist separatedly. ### dol.filesys.mk_dirs_if_missing_preset(self, k, v, , max_dirs_to_make=None, verbose=False) Preset function that will make the store create directories on write as needed. ### dol.filesys.mk_json_bytes_wrap(, loads_kwargs=None, dumps_kwargs=None) Make a `wrap_kvs` value-codec wrapper for JSON, with kwargs for `json.loads`/`json.dumps`. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) ### dol.filesys.mk_pickle_bytes_wrap(, loads_kwargs=None, dumps_kwargs=None) Make a `wrap_kvs` value-codec wrapper for pickle, with kwargs for `pickle.loads`/`pickle.dumps`. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) ### dol.filesys.mk_tmp_dol_dir(dirname='', make_it_if_necessary=True, verbose=False) Create and return a path to a temporary directory that’s guaranteed to be accessible to the user. * **Parameters:** * **dirname** – Optional subdirectory name to append to the temporary directory path * **make_it_if_necessary** – Whether to create the directory if it doesn’t exist * **verbose** – Controls verbosity when creating directories * **Returns:** Path to a temporary directory that the user has access to #### NOTE This function creates a user-specific temporary directory to avoid permission issues with system-wide temporary directories. ### dol.filesys.paths_in_dir(rootdir, include_hidden=False) Yield the paths of the entries of `rootdir` (directories with a trailing separator), skipping hidden ones unless `include_hidden`. ### dol.filesys.process_path(\*path, ensure_dir_exists=False, assert_exists=False, ensure_endswith_slash=False, ensure_does_not_end_with_slash=False, expanduser=True, expandvars=True, abspath=True, rootdir='') Process a path string, ensuring it exists, and optionally expanding user. * **Parameters:** * **path** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The path to process. Can be multiple components of a path. * **ensure_dir_exists** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to ensure the path exists. * **assert_exists** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to assert that the path exists. * **ensure_endswith_slash** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to ensure the path ends with a slash. * **ensure_does_not_end_with_slash** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to ensure the path does not end with a slash. * **expanduser** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to expand the user in the path. * **expandvars** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to expand environment variables in the path. * **abspath** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to convert the path to an absolute path. * **rootdir** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The root directory to prepend to the path. * **Returns:** The processed path. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) The result uses the running OS’s native separator, so these examples assert OS-independently (the literal forward-slash form is what you get on POSIX): ```pycon >>> import os >>> process_path('a', 'b', 'c').endswith(os.path.join('a', 'b', 'c')) True >>> p = process_path( ... 'a', 'b', 'c', rootdir='root_dir', ... ensure_endswith_slash=True, abspath=False, expanduser=False, expandvars=False, ... ) >>> p == os.path.join('root_dir', 'a', 'b', 'c') + os.sep True ``` ### dol.filesys.resolve_dir(dirpath, assert_existence=False, ensure_existence=False) Resolve a path to a full, real, path to a directory ### dol.filesys.resolve_path(path, assert_existence=False) Resolve a path to a full, real, (file or folder) path (opt assert existence). That is, resolve situations where ~ and . prefix the paths. ### dol.filesys.subfolder_stores(root_folder, \*, max_levels=None, include_hidden=False, relative_paths=True, slash_suffix=False, folder_to_store=) Create a store of subfolders of a given folder, where the keys are the subfolder paths (by default, relative and slash-less) and the values are stores of these subfolders. By default, all subfolders will be taken, recursively, but this can be controlled by the `max_levels` parameter. ### dol.filesys.temp_dir(dirname='', make_it_if_necessary=True, verbose=False) Create and return a path to a temporary directory that’s guaranteed to be accessible to the user. * **Parameters:** * **dirname** – Optional subdirectory name to append to the temporary directory path * **make_it_if_necessary** – Whether to create the directory if it doesn’t exist * **verbose** – Controls verbosity when creating directories * **Returns:** Path to a temporary directory that the user has access to #### NOTE This function creates a user-specific temporary directory to avoid permission issues with system-wide temporary directories. ### dol.filesys.validate_key_and_raise_key_error_on_exception(func) Method decorator: validate the key first, and re-raise any exception of the method as a `KeyError`. # _autosummary/dol.html.md # dol Core tools to build simple interfaces to complex data sources and bend the interface to your will (and need). `dol` wraps any storage backend (files, S3, databases, dicts) behind a dict-like interface, and transforms that interface with composable layers. Start with `wrap_kvs` (key/value transforms), the file stores (`Files`, `TextFiles`, `JsonFiles`, `PickleFiles`), the ready-made codecs (`ValueCodecs`, `KeyCodecs`), `filt_iter` (key filtering) and `cache_this` (caching). ```pycon >>> from dol import wrap_kvs >>> import json >>> s = wrap_kvs({}, obj_of_data=json.loads, data_of_obj=json.dumps) >>> s['a'] = {'x': 1} >>> s['a'], s.store ({'x': 1}, {'a': '{"x": 1}'}) ``` ### Functions | [`ihead`](_autosummary/dol.html.md#dol.ihead)(store[, n]) | Get the first item of an iterable, or a list of the first n items | |----------------------------------------------------------------------|---------------------------------------------------------------------| | [`kvhead`](_autosummary/dol.html.md#dol.kvhead)(store[, n]) | Get the first item of a kv store, or a list of the first n items | ### dol.ihead(store, n=1) Get the first item of an iterable, or a list of the first n items ### dol.kvhead(store, n=1) Get the first item of a kv store, or a list of the first n items ### Modules | [`appendable`](_autosummary/dol.appendable.html.md#dol.appendable)([store_cls, return_keys, ...]) | Makes a new class with append (and consequential extend) methods | |-------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------| | [`base`](_autosummary/dol.base.html.md#module-dol.base) | Base classes for making stores. | | [`caching`](_autosummary/dol.caching.html.md#module-dol.caching) | Tools to add caching layers to stores and methods. | | [`content`](_autosummary/dol.content.html.md#module-dol.content) | Content references and content-addressed storage — the flat "blob" layer. | | [`dig`](_autosummary/dol.dig.html.md#module-dol.dig) | Layers introspection: walk the layers of a wrapped store and trace a key through them. | | [`errors`](_autosummary/dol.errors.html.md#module-dol.errors) | Error objects and utils. | | [`explicit`](_autosummary/dol.explicit.html.md#module-dol.explicit) | Stores whose keys are given explicitly, with values fetched lazily from a source. | | [`filesys`](_autosummary/dol.filesys.html.md#module-dol.filesys) | File system access: dict-like stores over folders and files. | | [`kv_codecs`](_autosummary/dol.kv_codecs.html.md#module-dol.kv_codecs) | Tools to make Key-Value Codecs (encoder-decoder pairs) from standard library tools. | | [`misc`](_autosummary/dol.misc.html.md#module-dol.misc) | Functions to read from and write to misc sources, choosing the codec from the key. | | [`mixins`](_autosummary/dol.mixins.html.md#module-dol.mixins) | Mixins that add or restrict store behaviours. | | [`naming`](_autosummary/dol.naming.html.md#module-dol.naming) | This module is about generating, validating, and operating on (parametrized) fields (i.e. strings, e.g. paths). | | [`paths`](_autosummary/dol.paths.html.md#module-dol.paths) | Module for path (and path-like) object manipulation | | [`recipes`](_autosummary/dol.recipes.html.md#module-dol.recipes) | Recipes using dol | | [`signatures`](_autosummary/dol.signatures.html.md#module-dol.signatures) | Signature calculus: Tools to make it easier to work with function's signatures. | | [`sources`](_autosummary/dol.sources.html.md#module-dol.sources) | Key-value views of disparate sources. | | [`tools`](_autosummary/dol.tools.html.md#module-dol.tools) | Various tools to add functionality to stores. | | [`trans`](_autosummary/dol.trans.html.md#module-dol.trans) | Tools to wrap stores with key/value transforms, filters, caches and other layers. | | [`trash`](_autosummary/dol.trash.html.md#module-dol.trash) | Cross-platform file trash/recycle bin functionality for dol. | | [`util`](_autosummary/dol.util.html.md#module-dol.util) | General util objects: function composition, grouping, partial classes, file helpers. | | [`zipfiledol`](_autosummary/dol.zipfiledol.html.md#module-dol.zipfiledol) | Data object layers and other utils to work with zip files. | # _autosummary/dol.kv_codecs.html.md # dol.kv_codecs Tools to make Key-Value Codecs (encoder-decoder pairs) from standard library tools. A codec is a store wrapper: `ValueCodecs.json()` encodes values on write and decodes them on read, `KeyCodecs.suffixed('.json')` adds the suffix on the way in and strips it on the way out. Codecs compose with `+`. Main entry points: - `ValueCodecs`: ready-made value codecs (json, pickle, gzip, csv, str_to_bytes, …) - `KeyCodecs`: ready-made key codecs (suffixed, prefixed, …) - `key_based_value_trans`: a value codec chosen from the key ```pycon >>> from dol.kv_codecs import ValueCodecs, KeyCodecs >>> s = ValueCodecs.json()({}) >>> s['a'] = {'x': 1} >>> s.store, s['a'] ({'a': '{"x": 1}'}, {'x': 1}) >>> k = KeyCodecs.suffixed('.json')({}) >>> k['a'] = 1 >>> k.store, list(k) ({'a.json': 1}, ['a']) ``` ### Functions | [`add_invertible_key_decoder`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.add_invertible_key_decoder)(store, \*, decoder) | Add a key decoder to a store (instance) | |---------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------| | [`codec_wrap`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.codec_wrap)(cls, encoder, decoder, \*[, exclude]) | Make a `cls` codec factory from an `encoder` and a `decoder`, with the merged signature of both. | | [`common_prefix_keys_wrap`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.common_prefix_keys_wrap)(s) | Transforms keys of mapping to omit the longest prefix they have in common | | [`csv_decode`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.csv_decode)(string[, dialect, delimiter, ...]) | Decode a CSV string into a list of rows (`csv.reader` arguments accepted). | | [`csv_dict_decode`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.csv_dict_decode)(string, fieldnames[, ...]) | Decode a csv string into a list of dicts. | | [`csv_dict_encode`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.csv_dict_encode)(string, fieldnames[, ...]) | Encode a list of dicts into a csv string. | | [`csv_encode`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.csv_encode)(string[, dialect, delimiter, ...]) | Encode rows (an iterable of iterables) into a CSV string (`csv.writer` arguments accepted). | | [`extract_arguments`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.extract_arguments)(func, args, kwargs) | Map `args`/`kwargs` to `func`'s parameter names, leniently (partial and excess allowed, kinds ignored). | | [`key_based_codec_factory`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.key_based_codec_factory)(key_mapping[, key_func]) | A factory that creates a key codec that uses the key to determine the codec to use. | | [`key_based_value_trans`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.key_based_value_trans)(key_func, ...[, k]) | A factory that creates a value codec that uses the key to determine the codec to use. | ### Classes | [`CodecCollection`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.CodecCollection)(\*args, \*\*kwargs) | The base class for collections of codecs. | |----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------| | [`KeyCodecs`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.KeyCodecs)(\*args, \*\*kwargs) | A collection of key codecs | | [`KeyValueCodecs`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.KeyValueCodecs)(\*args, \*\*kwargs) | A collection of key-value codecs that can be used with postget and preset kv_wraps. | | [`NotGiven`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.NotGiven)() | A singleton to indicate that a value was not given | | [`ValueCodecs`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.ValueCodecs)(\*args, \*\*kwargs) | A collection of value codec factories using standard lib tools. | ### *class* dol.kv_codecs.CodecCollection(\*args, \*\*kwargs) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The base class for collections of codecs. Makes sure that the class cannot be instantiated, but only used as a collection. Also provides an \_iter_codecs method that iterates over the codec names. ### *class* dol.kv_codecs.KeyCodecs(\*args, \*\*kwargs) Bases: [`CodecCollection`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.CodecCollection) A collection of key codecs #### mapped_keys(decoder=None) A factory that creates a key codec that uses “explicit” mappings to encode and decode keys. The encoders and decoders can be an explicit mapping of a function. If the encoder is a mapping, the decoder is the inverse of that mapping. If given explicitly, this will be asserted. If not, the decoder will be computed by swapping the keys and values of the encoder and asserting that no values were lost in the process (that is, that the mappings are invertible). The statements above are true if you swap “encoder” and “decoder”. ```pycon >>> km = KeyCodecs.mapped_keys({'a': 1, 'b': 2}) >>> km.encoder('a') 1 >>> km.decoder(1) 'a' ``` If the encoder is a function, the decoder must be an iterable of keys who will be used as arguments of the function to get the encoded key, and the decode will be the inverse of that mapping. The statement above is true if you swap “encoder” and “decoder”. ```pycon >>> km = KeyCodecs.mapped_keys(['a', 'b'], str.upper) >>> km.encoder('A') 'a' >>> km.decoder('a') 'A' ``` ### *class* dol.kv_codecs.KeyValueCodecs(\*args, \*\*kwargs) Bases: [`CodecCollection`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.CodecCollection) A collection of key-value codecs that can be used with postget and preset kv_wraps. ### *class* dol.kv_codecs.NotGiven Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A singleton to indicate that a value was not given ### *class* dol.kv_codecs.ValueCodecs(\*args, \*\*kwargs) Bases: [`CodecCollection`](_autosummary/dol.kv_codecs.html.md#dol.kv_codecs.CodecCollection) A collection of value codec factories using standard lib tools. ```pycon >>> json_codec = ValueCodecs.json() # call the json codec factory >>> encoder, decoder = json_codec >>> encoder({'b': 2}) '{"b": 2}' >>> decoder('{"b": 2}') {'b': 2} ``` The `json_codec` object is also a `Mapping` value wrapper: ```pycon >>> backend = dict() >>> interface = json_codec(backend) >>> interface['a'] = {'b': 2} # we write a dict >>> assert backend == {'a': '{"b": 2}'} # json was written in backend >>> interface['a'] # but this json is decoded to a dict when read from interface {'b': 2} ``` In order not to have to call the codec factory when you just want the default, we’ve made a `default` attribute that contains all the default codecs: ```pycon >>> backend = dict() >>> interface = ValueCodecs.default.json(backend) >>> interface['a'] = {'b': 2} # we write a dict >>> assert backend == {'a': '{"b": 2}'} # json was written in backend ``` For times when you want to parametrize your code though, know that you can also pass arguments to the encoder and decoder when you make your codec. For example, to make a json codec that indents the json, you can do: ```pycon >>> json_codec = ValueCodecs.json(indent=2) >>> backend = dict() >>> interface = json_codec(backend) >>> interface['a'] = {'b': 2} # we write a dict >>> print(backend['a']) # written in backend with indent { "b": 2 } ``` #### b64 *= * #### *class* default Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) To contain default codecs. Is populated by @_add_default_codecs #### io *= * #### *class* methodcaller(name, , \*args, \*\*kwargs) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Return a callable object that calls the given method on its operand. After f = methodcaller(‘name’), the call f(r) returns r.name(). After g = methodcaller(‘name’, ‘date’, foo=1), the call g(r) returns r.name(‘date’, foo=1). #### single_nested_value() ```pycon >>> d = { ... 1: {'en': 'one', 'fr': 'un', 'sp': 'uno'}, ... 2: {'en': 'two', 'fr': 'deux', 'sp': 'dos'}, ... } >>> en = ValueCodecs.single_nested_value('en')(d) >>> en[1] 'one' >>> en[1] = 'ONE' >>> d[1] # note that here d[1] is completely replaced (not updated) {'en': 'ONE'} ``` #### tar_compress(file_name='data.bin') Bytes of an (uncompressed) tar archive holding `data_bytes` as a single file. ```pycon >>> tar_decompress(tar_compress(b'hello', file_name='x.bin')) b'hello' ``` #### tar_decompress() Bytes of the first file found in the tar archive `tar_bytes` (None if none). #### tuple_of_dict() Get a tuple-view of dict values. ```pycon >>> d = { ... 1: {'en': 'one', 'fr': 'un', 'sp': 'uno'}, ... 2: {'en': 'two', 'fr': 'deux', 'sp': 'dos'}, ... } >>> codec = ValueCodecs.tuple_of_dict(['fr', 'sp']) >>> codec.encoder(['deux', 'tre']) {'fr': 'deux', 'sp': 'tre'} >>> codec.decoder({'en': 'one', 'fr': 'un', 'sp': 'uno'}) ('un', 'uno') >>> frsp = codec(d) >>> frsp[2] ('deux', 'dos') >>> ('deux', 'dos') ('deux', 'dos') >>> frsp[2] = ('DEUX', 'DOS') >>> frsp[2] ('DEUX', 'DOS') ``` Note that writes completely replace the values in the backend dict, it doesn’t update them: ```pycon >>> d[2] {'fr': 'DEUX', 'sp': 'DOS'} ``` See also `dol.KeyTemplate` for more general key-based views. #### zip_compress(filename='some_bytes', , compression=8, allowZip64=True, compresslevel=None, strict_timestamps=True, encoding='utf-8') Compress input bytes, returning the compressed bytes * **Return type:** [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes) ```pycon >>> b = b'x' * 1000 + b'y' * 1000 # 2000 (quite compressible) bytes >>> len(b) 2000 >>> >>> zipped_bytes = zip_compress(b) >>> # Note: Compression details will be system dependent >>> len(zipped_bytes) 137 >>> unzipped_bytes = zip_decompress(zipped_bytes) >>> unzipped_bytes == b # verify that unzipped bytes are the same as the original True >>> >>> from dol.zipfiledol import compression_methods >>> >>> zipped_bytes = zip_compress(b, compression=compression_methods['bzip2']) >>> # Note: Compression details will be system dependent >>> len(zipped_bytes) 221 >>> unzipped_bytes = zip_decompress(zipped_bytes) >>> unzipped_bytes == b # verify that unzipped bytes are the same as the original True ``` #### zip_decompress(, allowZip64=True, compresslevel=None, strict_timestamps=True) Decompress input bytes of a single file zip, returning the uncompressed bytes See `zip_compress` for usage examples. * **Return type:** [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes) ### dol.kv_codecs.add_invertible_key_decoder(store, , decoder) Add a key decoder to a store (instance) ### dol.kv_codecs.codec_wrap(cls, encoder, decoder, , exclude=()) Make a `cls` codec factory from an `encoder` and a `decoder`, with the merged signature of both. ### dol.kv_codecs.common_prefix_keys_wrap(s) Transforms keys of mapping to omit the longest prefix they have in common ### dol.kv_codecs.csv_decode(string, dialect='excel', delimiter=',', quotechar='"', escapechar=None, doublequote=True, skipinitialspace=False, lineterminator='\\\\r\\\\n', quoting=0, strict=False) Decode a CSV string into a list of rows (`csv.reader` arguments accepted). ### dol.kv_codecs.csv_dict_decode(string, fieldnames, dialect='excel', delimiter=',', quotechar='"', escapechar=None, doublequote=True, skipinitialspace=False, lineterminator='\\\\r\\\\n', quoting=0, strict=False, restkey=None, restval='', extrasaction='raise', fieldcasts=None) Decode a csv string into a list of dicts. * **Parameters:** * **string** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The csv string to decode * **fieldcasts** – A function that takes a row and returns a row with the same keys but with values cast to the desired type. If a dict, it should be a mapping from fieldnames to cast functions. If an iterable, it should be an iterable of cast functions, in which case each cast function will be applied to each element of the row, element wise. ```pycon >>> data = [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}] >>> encoded = csv_dict_encode(data, fieldnames=['a', 'b']) >>> encoded 'a,b\r\n1,2\r\n3,4\r\n' >>> csv_dict_decode(encoded) [{'a': '1', 'b': '2'}, {'a': '3', 'b': '4'}] ``` See that you don’t get back when you started with. The ints aren’t ints anymore! You can resolve this by using the fieldcasts argument (that’s our argument – not present in builtin csv module). I should be a function (that transforms a dict to the one you want) or list or tuple of the same size as the row (that specifies the cast function for each field) ```pycon >>> csv_dict_decode(encoded, fieldnames=['a', 'b'], fieldcasts=[int] * 2) [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}] >>> csv_dict_decode(encoded, fieldnames=['a', 'b'], fieldcasts={'b': float}) [{'a': '1', 'b': 2.0}, {'a': '3', 'b': 4.0}] ``` ### dol.kv_codecs.csv_dict_encode(string, fieldnames, dialect='excel', delimiter=',', quotechar='"', escapechar=None, doublequote=True, skipinitialspace=False, lineterminator='\\\\r\\\\n', quoting=0, strict=False, restkey=None, restval='', extrasaction='raise', fieldcasts=None) Encode a list of dicts into a csv string. ```pycon >>> data = [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}] >>> encoded = csv_dict_encode(data, fieldnames=['a', 'b']) >>> encoded 'a,b\r\n1,2\r\n3,4\r\n' ``` ### dol.kv_codecs.csv_encode(string, dialect='excel', delimiter=',', quotechar='"', escapechar=None, doublequote=True, skipinitialspace=False, lineterminator='\\\\r\\\\n', quoting=0, strict=False) Encode rows (an iterable of iterables) into a CSV string (`csv.writer` arguments accepted). ### dol.kv_codecs.extract_arguments(func, args, kwargs) Map `args`/`kwargs` to `func`’s parameter names, leniently (partial and excess allowed, kinds ignored). ### dol.kv_codecs.key_based_codec_factory(key_mapping, key_func=) A factory that creates a key codec that uses the key to determine the codec to use. ### dol.kv_codecs.key_based_value_trans(key_func, value_trans_mapping, default_factory, k=) A factory that creates a value codec that uses the key to determine the codec to use. Below, `key_func` gets the extension of a file path: ```pycon >>> import json >>> from functools import partial >>> key_func = lambda k: os.path.splitext(k)[1] >>> value_trans_mapping = {'.json': json.loads, '.txt': bytes.decode} >>> default_factory = partial(ValueError, "No codec for this extension") >>> trans = key_based_value_trans( ... key_func, value_trans_mapping, default_factory=lambda: identity_func ... ) ``` ### dol.kv_codecs.key_value_wrap(encoder, decoder, , exclude=()) Make a `cls` codec factory from an `encoder` and a `decoder`, with the merged signature of both. ### dol.kv_codecs.key_wrap(encoder, decoder, , exclude=()) Make a `cls` codec factory from an `encoder` and a `decoder`, with the merged signature of both. ### dol.kv_codecs.value_wrap(encoder, decoder, , exclude=()) Make a `cls` codec factory from an `encoder` and a `decoder`, with the merged signature of both. # _autosummary/dol.misc.html.md # dol.misc Functions to read from and write to misc sources, choosing the codec from the key. `get_obj`/`set_obj` read and write a file with the codec picked from its extension (`.json`, `.csv`, `.pkl`, …); `MiscReaderMixin`/`MiscStoreMixin` add the same key-conditioned (de)serialization to any store. ```pycon >>> from dol.misc import MiscStoreMixin >>> class M(MiscStoreMixin, dict): ... pass >>> m = M() >>> m['a.json'] = {'x': 1} >>> dict.__getitem__(m, 'a.json'), m['a.json'] (b'{"x": 1}', {'x': 1}) ``` ### Functions | `csv_fileobj`(csv_data, \*args, \*\*kwargs) | | |------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------| | `dflt_dflt_incoming_val_trans`(x) | | | `dflt_func_key`(self, k) | | | [`get_obj`](_autosummary/dol.misc.html.md#dol.misc.get_obj)(k[, store, ...]) | A quick way to get an object, with default. | | `identity_method`(x) | | | [`set_obj`](_autosummary/dol.misc.html.md#dol.misc.set_obj)(k, v[, store, ...]) | A quick way to set an object, with defaults for everything (but the key and value, you know, a clue of what you want to store). | | `url_to_bytes`(url) | | ### Classes | [`MiscGetter`](_autosummary/dol.misc.html.md#dol.misc.MiscGetter)([store, ...]) | An object to write (and only write) to a store (default local files) with automatic deserialization according to a property of the key (default: file extension). | |----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`MiscGetterAndSetter`](_autosummary/dol.misc.html.md#dol.misc.MiscGetterAndSetter)([store, ...]) | An object to read and write (and nothing else) to a store (default local) with automatic (de)serialization according to a property of the key (default: file extension). | | [`MiscReaderMixin`](_autosummary/dol.misc.html.md#dol.misc.MiscReaderMixin)([...]) | Mixin to transform incoming vals according to the key their under. | | [`MiscStoreMixin`](_autosummary/dol.misc.html.md#dol.misc.MiscStoreMixin)([incoming_val_trans_for_key, ...]) | Mixin to transform incoming and outgoing vals according to the key their under. | ### *class* dol.misc.MiscGetter(store=Files(rootdir='', subpath='', pattern_for_field=None, max_levels=None, include_hidden=False, assert_rootdir_existence=False), incoming_val_trans_for_key={'.bin': , '.csv': >, '.gz': , '.gzip': , '.json': >, '.pickle': >, '.pkl': >, '.txt': >, '.zip': }, dflt_incoming_val_trans=, func_key=>) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An object to write (and only write) to a store (default local files) with automatic deserialization according to a property of the key (default: file extension). ```pycon >>> from dol.misc import get_obj, misc_objs_get >>> import os >>> import json >>> >>> pjoin = lambda *p: os.path.join(os.path.expanduser('~'), *p) >>> path = pjoin('tmp.json') >>> d = {'a': {'b': {'c': [1, 2, 3]}}} >>> json.dump(d, open(path, 'w')) # putting a json file there, the normal way, so we can use it later >>> >>> k = path >>> t = get_obj(k) # if you'd like to use a function >>> assert t == d >>> tt = misc_objs_get[k] # if you'd like to use an object (note: can get, but nothing else (no list, set, del, etc)) >>> assert tt == d >>> t {'a': {'b': {'c': [1, 2, 3]}}} ``` ### *class* dol.misc.MiscGetterAndSetter(store=Files(rootdir='', subpath='', pattern_for_field=None, max_levels=None, include_hidden=False, assert_rootdir_existence=False), incoming_val_trans_for_key={'.bin': , '.csv': >, '.gz': , '.gzip': , '.json': >, '.pickle': >, '.pkl': >, '.txt': >, '.zip': }, outgoing_val_trans_for_key={'.bin': , '.cnf': >, '.conf': >, '.config': >, '.csv': , '.gz': , '.gzip': , '.ini': >, '.json': >, '.pickle': >, '.pkl': >, '.txt': >}, dflt_incoming_val_trans=, func_key=>) Bases: [`MiscGetter`](_autosummary/dol.misc.html.md#dol.misc.MiscGetter) An object to read and write (and nothing else) to a store (default local) with automatic (de)serialization according to a property of the key (default: file extension). ```pycon >>> from dol.misc import set_obj, misc_objs # the function and the object >>> import json >>> import os >>> >>> pjoin = lambda *p: os.path.join(os.path.expanduser('~'), *p) >>> >>> d = {'a': {'b': {'c': [1, 2, 3]}}} >>> misc_objs[pjoin('tmp.json')] = d >>> filepath = os.path.expanduser('~/tmp.json') >>> assert misc_objs[filepath] == d # yep, it's there, and can be retrieved >>> assert json.load(open(filepath)) == d # in case you don't believe it's an actual json file >>> >>> # using pickle >>> misc_objs[pjoin('tmp.pkl')] = d >>> assert misc_objs[pjoin('tmp.pkl')] == d >>> >>> # using txt >>> misc_objs[pjoin('tmp.txt')] = 'hello world!' >>> assert misc_objs[pjoin('tmp.txt')] == 'hello world!' >>> >>> # using csv >>> misc_objs[pjoin('tmp.csv')] = [[1,2,3], ['a','b','c']] >>> assert misc_objs[pjoin('tmp.csv')] == [['1','2','3'], ['a','b','c']] # yeah, well, not numbers, but you deal with it >>> >>> # using bin ... misc_objs[pjoin('tmp.bin')] = b'let us pretend these are bytes of an audio waveform' >>> assert misc_objs[pjoin('tmp.bin')] == b'let us pretend these are bytes of an audio waveform' ``` ### *class* dol.misc.MiscReaderMixin(incoming_val_trans_for_key=None, dflt_incoming_val_trans=None, func_key=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mixin to transform incoming vals according to the key their under. #### WARNING If used as a subclass, this mixin should (in general) be placed before the store ```pycon >>> # make a reader that will wrap a dict >>> class MiscReader(MiscReaderMixin, dict): ... def __init__(self, d, ... incoming_val_trans_for_key=None, ... dflt_incoming_val_trans=None, ... func_key=None): ... dict.__init__(self, d) ... MiscReaderMixin.__init__(self, incoming_val_trans_for_key, dflt_incoming_val_trans, func_key) ... >>> >>> incoming_val_trans_for_key = dict( ... MiscReaderMixin._incoming_val_trans_for_key, # take the existing defaults... ... **{'.bin': lambda v: [ord(x) for x in v.decode()], # ... override how to handle the .bin extension ... '.reverse_this': lambda v: v[::-1] # add a new extension (and how to handle it) ... }) >>> >>> import pickle >>> d = { ... 'a.bin': b'abc123', ... 'a.reverse_this': b'abc123', ... 'a.csv': b'event,year\n Magna Carta,1215\n Guido,1956', ... 'a.txt': b'this is not a text', ... 'a.pkl': pickle.dumps(['text', [str, map], {'a list': [1, 2, 3]}]), ... 'a.json': '{"str": "field", "int": 42, "float": 3.14, "array": [1, 2], "nested": {"a": 1, "b": 2}}', ... } >>> >>> s = MiscReader(d=d, incoming_val_trans_for_key=incoming_val_trans_for_key) >>> list(s) ['a.bin', 'a.reverse_this', 'a.csv', 'a.txt', 'a.pkl', 'a.json'] >>> s['a.bin'] [97, 98, 99, 49, 50, 51] >>> s['a.reverse_this'] b'321cba' >>> s['a.csv'] [['event', 'year'], [' Magna Carta', '1215'], [' Guido', '1956']] >>> s['a.pkl'] ['text', [, ], {'a list': [1, 2, 3]}] >>> s['a.json'] {'str': 'field', 'int': 42, 'float': 3.14, 'array': [1, 2], 'nested': {'a': 1, 'b': 2}} ``` ### *class* dol.misc.MiscStoreMixin(incoming_val_trans_for_key=None, outgoing_val_trans_for_key=None, dflt_incoming_val_trans=None, dflt_outgoing_val_trans=None, func_key=None) Bases: [`MiscReaderMixin`](_autosummary/dol.misc.html.md#dol.misc.MiscReaderMixin) Mixin to transform incoming and outgoing vals according to the key their under. #### WARNING If used as a subclass, this mixin should (in general) be placed before the store #### SEE ALSO preset and postget args from wrap_kvs decorator from dol.trans. ```pycon >>> # Make a class to wrap a dict with a layer that transforms written and read values >>> class MiscStore(MiscStoreMixin, dict): ... def __init__(self, d, ... incoming_val_trans_for_key=None, outgoing_val_trans_for_key=None, ... dflt_incoming_val_trans=None, dflt_outgoing_val_trans=None, ... func_key=None): ... dict.__init__(self, d) ... MiscStoreMixin.__init__(self, incoming_val_trans_for_key, outgoing_val_trans_for_key, ... dflt_incoming_val_trans, dflt_outgoing_val_trans, func_key) ... >>> >>> outgoing_val_trans_for_key = dict( ... MiscStoreMixin._outgoing_val_trans_for_key, # take the existing defaults... ... **{'.bin': lambda v: ''.join([chr(x) for x in v]).encode(), # ... override how to handle the .bin extension ... '.reverse_this': lambda v: v[::-1] # add a new extension (and how to handle it) ... }) >>> ss = MiscStore(d={}, # store starts empty ... incoming_val_trans_for_key={}, # overriding incoming trans so we can see the raw data later ... outgoing_val_trans_for_key=outgoing_val_trans_for_key) ... >>> # here's what we're going to write in the store >>> data_to_write = { ... 'a.bin': [97, 98, 99, 49, 50, 51], ... 'a.reverse_this': b'321cba', ... 'a.csv': [['event', 'year'], [' Magna Carta', '1215'], [' Guido', '1956']], ... 'a.txt': 'this is not a text', ... 'a.pkl': ['text', [str, map], {'a list': [1, 2, 3]}], ... 'a.json': {'str': 'field', 'int': 42, 'float': 3.14, 'array': [1, 2], 'nested': {'a': 1, 'b': 2}}} >>> # write this data in our store >>> for k, v in data_to_write.items(): ... ss[k] = v >>> list(ss) ['a.bin', 'a.reverse_this', 'a.csv', 'a.txt', 'a.pkl', 'a.json'] >>> # Looking at the contents (what was actually stored/written) >>> for k, v in ss.items(): ... if k != 'a.pkl': ... print(f"{k}: {v}") ... else: # need to verify pickle data differently, since printing contents is problematic in doctest ... assert pickle.loads(v) == data_to_write['a.pkl'] a.bin: b'abc123' a.reverse_this: b'abc123' a.csv: b'event,year\r\n Magna Carta,1215\r\n Guido,1956\r\n' a.txt: b'this is not a text' a.json: b'{"str": "field", "int": 42, "float": 3.14, "array": [1, 2], "nested": {"a": 1, "b": 2}}' ``` ### dol.misc.get_obj(k, store=Files(rootdir='', subpath='', pattern_for_field=None, max_levels=None, include_hidden=False, assert_rootdir_existence=False), incoming_val_trans_for_key={'.bin': , '.csv': >, '.gz': , '.gzip': , '.json': >, '.pickle': >, '.pkl': >, '.txt': >, '.zip': }, dflt_incoming_val_trans=, func_key=>) A quick way to get an object, with default… everything (but the key, you know, a clue of what you want) ### dol.misc.set_obj(k, v, store=Files(rootdir='', subpath='', pattern_for_field=None, max_levels=None, include_hidden=False, assert_rootdir_existence=False), outgoing_val_trans_for_key={'.bin': , '.cnf': >, '.conf': >, '.config': >, '.csv': , '.gz': , '.gzip': , '.ini': >, '.json': >, '.pickle': >, '.pkl': >, '.txt': >}, func_key=>) A quick way to set an object, with defaults for everything (but the key and value, you know, a clue of what you want to store). # _autosummary/dol.mixins.html.md # dol.mixins Mixins that add or restrict store behaviours. Main entry points: - `ReadOnlyMixin`: forbid writes and deletes - `OverWritesNotAllowedMixin`: forbid writing to an existing key - `SimpleJsonMixin`: JSON-encoded values - `IterBasedSizedContainerMixin`: `__len__` and `__contains__` from `__iter__` ```pycon >>> from dol.mixins import OverWritesNotAllowedMixin >>> class P(OverWritesNotAllowedMixin, dict): ... pass >>> p = P() >>> p['a'] = 1 >>> p['a'] = 2 Traceback (most recent call last): ... dol.errors.OverWritesNotAllowedError: key a already exists and cannot be overwritten... ``` ### Classes | [`FilteredKeysMixin`](_autosummary/dol.mixins.html.md#dol.mixins.FilteredKeysMixin)() | Filters \_\_iter_\_ and \_\_contains_\_ with (the boolean filter function attribute) \_key_filt. | |---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------| | `GetBasedContainerMixin`() | | | `HashableMixin`() | | | [`IdentityKeysWrapMixin`](_autosummary/dol.mixins.html.md#dol.mixins.IdentityKeysWrapMixin)() | Transparent KeysWrapABC. | | [`IdentityKvWrapMixin`](_autosummary/dol.mixins.html.md#dol.mixins.IdentityKvWrapMixin)() | Transparent Keys and Vals Wrap | | [`IdentityValsWrapMixin`](_autosummary/dol.mixins.html.md#dol.mixins.IdentityValsWrapMixin)() | Transparent ValsWrapABC. | | `IterBasedContainerMixin`() | | | [`IterBasedSizedContainerMixin`](_autosummary/dol.mixins.html.md#dol.mixins.IterBasedSizedContainerMixin)() | An ABC that defines: | | `IterBasedSizedMixin`() | | | [`OverWritesNotAllowedMixin`](_autosummary/dol.mixins.html.md#dol.mixins.OverWritesNotAllowedMixin)() | Mixin for only allowing a write to a key if they key doesn't already exist. | | [`ReadOnlyMixin`](_autosummary/dol.mixins.html.md#dol.mixins.ReadOnlyMixin)() | Put this as your first parent class to disallow write/delete operations | | [`SimpleJsonMixin`](_autosummary/dol.mixins.html.md#dol.mixins.SimpleJsonMixin)() | simple json serialization. | | [`StringKvWrap`](_autosummary/dol.mixins.html.md#dol.mixins.StringKvWrap)() | | ### *class* dol.mixins.FilteredKeysMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Filters \_\_iter_\_ and \_\_contains_\_ with (the boolean filter function attribute) \_key_filt. ### *class* dol.mixins.IdentityKeysWrapMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Transparent KeysWrapABC. Often placed in the mro to satisfy the KeysWrapABC need in a neutral way. This is useful in cases where the keys the persistence functions work with are the same as those you want to work with. ### *class* dol.mixins.IdentityKvWrapMixin Bases: [`IdentityKeysWrapMixin`](_autosummary/dol.mixins.html.md#dol.mixins.IdentityKeysWrapMixin), [`IdentityValsWrapMixin`](_autosummary/dol.mixins.html.md#dol.mixins.IdentityValsWrapMixin) Transparent Keys and Vals Wrap ### *class* dol.mixins.IdentityValsWrapMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Transparent ValsWrapABC. Often placed in the mro to satisfy the KeysWrapABC need in a neutral way. This is useful in cases where the values can be persisted by \_\_setitem_\_ as is (or the serialization is handled somewhere in the \_\_setitem_\_ method. ### *class* dol.mixins.IterBasedSizedContainerMixin Bases: `IterBasedSizedMixin`, `IterBasedContainerMixin` An ABC that defines: 1. how to iterate over a collection of elements (keys) (`__iter__`) 2. check that a key is contained in the collection (`__contains__`), and 3. how to get the number of elements in the collection (`__len__`) This is exactly what the collections.abc.Collection (from which Keys inherits) does. The difference here, besides the “Keys” purpose-explicit name, is that Keys offers default `__len__` and `__contains__` definitions based on what ever `__iter__` the concrete class defines. Keys is a collection (i.e. a Sized (has \_\_len_\_), Iterable (has \_\_iter_\_), Container (has \_\_contains_\_). It’s purpose is to serve as a collection of object identifiers in a key->obj mapping. The Keys class doesn’t implement \_\_iter_\_ (so needs to be subclassed with a concrete class), but offers mixin \_\_len_\_ and \_\_contains_\_ methods based on a given \_\_iter_\_ method. Note that usually \_\_len_\_ and \_\_contains_\_ should be overridden to more, context-dependent, efficient methods. ### *class* dol.mixins.OverWritesNotAllowedMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mixin for only allowing a write to a key if they key doesn’t already exist. #### NOTE Should be before the persister in the MRO. ```pycon >>> class TestPersister(OverWritesNotAllowedMixin, dict): ... pass >>> p = TestPersister() >>> p['foo'] = 'bar' >>> #p['foo'] = 'bar2' # will raise error >>> p['foo'] = 'this value should not be stored' Traceback (most recent call last): ... dol.errors.OverWritesNotAllowedError: key foo already exists and cannot be overwritten. If you really want to write to that key, delete it before writing >>> p['foo'] # foo is still bar 'bar' >>> del p['foo'] >>> p['foo'] = 'this value WILL be stored' >>> p['foo'] 'this value WILL be stored' ``` ### *class* dol.mixins.ReadOnlyMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Put this as your first parent class to disallow write/delete operations ### *class* dol.mixins.SimpleJsonMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) simple json serialization. Useful to store and retrieve ### *class* dol.mixins.StringKvWrap Bases: [`IdentityKvWrapMixin`](_autosummary/dol.mixins.html.md#dol.mixins.IdentityKvWrapMixin) # _autosummary/dol.naming.html.md # dol.naming This module is about generating, validating, and operating on (parametrized) fields (i.e. strings, e.g. paths). Main entry points: - `StrTupleDict`: convert a templated name between string, tuple and dict forms - `mk_pattern_from_template_and_format_dict`: a compiled regex from a template - `get_fields_from_template`: the field names of a template ```pycon >>> from dol.naming import get_fields_from_template >>> get_fields_from_template('this{is}an{example}') ['is', 'example'] ``` ### Functions | [`dict_to_namedtuple`](_autosummary/dol.naming.html.md#dol.naming.dict_to_namedtuple)(d[, namedtuple_obj]) | | |----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`get_fields_from_template`](_autosummary/dol.naming.html.md#dol.naming.get_fields_from_template)(template) | Get list from {item} items of template string | | `mk_capture_patterns`(mapping_dict) | | | `mk_extract_pattern`(template[, format_dict, ...]) | | | `mk_format_mapping_dict`(format_dict, ...[, sep]) | | | [`mk_kwargs_trans`](_autosummary/dol.naming.html.md#dol.naming.mk_kwargs_trans)(\*\*trans_func_for_key) | Make a dict transformer from functions that depends solely on keys (of the dict to be transformed) Used to easily make process_kwargs and process_info_dict arguments for LinearNaming. | | `mk_named_capture_patterns`(mapping_dict) | | | [`mk_pattern_from_template_and_format_dict`](_autosummary/dol.naming.html.md#dol.naming.mk_pattern_from_template_and_format_dict)(...) | Make a compiled regex to match template | | `mk_prefix_templates_dicts`(template) | | | [`mk_store_from_path_format_store_cls`](_autosummary/dol.naming.html.md#dol.naming.mk_store_from_path_format_store_cls)([store, ...]) | Wrap a store (instance or class) that uses string keys to make it into a store that uses a specific key format. | | [`mk_tupled_store_from_path_format_store_cls`](_autosummary/dol.naming.html.md#dol.naming.mk_tupled_store_from_path_format_store_cls)([...]) | Wrap a store (instance or class) that uses string keys to make it into a store that uses a specific key format. | | [`namedtuple_to_dict`](_autosummary/dol.naming.html.md#dol.naming.namedtuple_to_dict)(nt) | | | [`template_to_pattern`](_autosummary/dol.naming.html.md#dol.naming.template_to_pattern)(mapping_dict, template) | Weave a `{field}` template into a regex, substituting each field with its capture pattern and **regex-escaping the literal text between fields**. | | [`update_fields_of_namedtuple`](_autosummary/dol.naming.html.md#dol.naming.update_fields_of_namedtuple)(nt, \*[, ...]) | Replace fields of namedtuple | | [`validate_kwargs`](_autosummary/dol.naming.html.md#dol.naming.validate_kwargs)(kwargs_to_validate, ...[, ...]) | Utility to validate a dict. | ### Classes | [`BigDocTest`](_autosummary/dol.naming.html.md#dol.naming.BigDocTest)() | Naming-scheme example holder whose (large) doctest is currently disabled. | |---------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| | [`KeyMapNames`](_autosummary/dol.naming.html.md#dol.naming.KeyMapNames) | | | [`KeyMaps`](_autosummary/dol.naming.html.md#dol.naming.KeyMaps)(key_of_id, id_of_key) | | | [`LinearNaming`](_autosummary/dol.naming.html.md#dol.naming.LinearNaming) | | | `NamingInterface`([params, validation_funs, ...]) | | | [`ParametricKeyStore`](_autosummary/dol.naming.html.md#dol.naming.ParametricKeyStore)(store[, keymap]) | | | [`PartialFormatter`](_autosummary/dol.naming.html.md#dol.naming.PartialFormatter)() | A string formatter that won't complain if the fields are only partially formatted. | | [`StoreWithDictKeys`](_autosummary/dol.naming.html.md#dol.naming.StoreWithDictKeys)(store[, keymap]) | | | [`StoreWithNamedTupleKeys`](_autosummary/dol.naming.html.md#dol.naming.StoreWithNamedTupleKeys)(store[, keymap]) | | | [`StoreWithTupleKeys`](_autosummary/dol.naming.html.md#dol.naming.StoreWithTupleKeys)(store[, keymap]) | | | [`StrTupleDict`](_autosummary/dol.naming.html.md#dol.naming.StrTupleDict)(template[, format_dict, ...]) | Convert a parametrized name between its string, tuple and dict forms. | | [`StrTupleDictWithPrefix`](_autosummary/dol.naming.html.md#dol.naming.StrTupleDictWithPrefix)(template[, ...]) | Converting from and to strings, tuples, and dicts, but with partial "prefix" specs allowed. | ### *class* dol.naming.BigDocTest Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Naming-scheme example holder whose (large) doctest is currently disabled. The former doctest is kept as comments in the class body. ### dol.naming.KeyMapNames alias of [`KeyMaps`](_autosummary/dol.naming.html.md#dol.naming.KeyMaps) ### *class* dol.naming.KeyMaps(key_of_id, id_of_key) Bases: [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) #### id_of_key Alias for field number 1 #### key_of_id Alias for field number 0 ### dol.naming.LinearNaming alias of [`StrTupleDictWithPrefix`](_autosummary/dol.naming.html.md#dol.naming.StrTupleDictWithPrefix) ### *class* dol.naming.ParametricKeyStore(store, keymap=None) Bases: [`Store`](_autosummary/dol.base.html.md#dol.base.Store) ### *class* dol.naming.PartialFormatter Bases: [`Formatter`](https://docs.python.org/3/library/string.html#string.Formatter) A string formatter that won’t complain if the fields are only partially formatted. But note that you will lose the spec part of your template (e.g. in {foo:1.2f}, you’ll loose the 1.2f if not foo is given – but {foo} will remain). ```pycon >>> partial_formatter = PartialFormatter() >>> str_template = 'foo:{foo} bar={bar} a={a} b={b:0.02f} c={c}' >>> partial_formatter.format(str_template, bar="BAR", b=34) 'foo:{foo} bar=BAR a={a} b=34.00 c={c}' ``` #### NOTE If you only need a formatting function (not the transformed formatting string), a simpler solution may be: ```python import functools format_str = functools.partial(str_template.format, bar="BAR", b=34) ``` See [https://stackoverflow.com/questions/11283961/partial-string-formatting](https://stackoverflow.com/questions/11283961/partial-string-formatting) for more options and discussions. ### *class* dol.naming.StoreWithDictKeys(store, keymap=None) Bases: [`ParametricKeyStore`](_autosummary/dol.naming.html.md#dol.naming.ParametricKeyStore) ### *class* dol.naming.StoreWithNamedTupleKeys(store, keymap=None) Bases: [`ParametricKeyStore`](_autosummary/dol.naming.html.md#dol.naming.ParametricKeyStore) ### *class* dol.naming.StoreWithTupleKeys(store, keymap=None) Bases: [`ParametricKeyStore`](_autosummary/dol.naming.html.md#dol.naming.ParametricKeyStore) ### *class* dol.naming.StrTupleDict(template, format_dict=None, process_kwargs=None, process_info_dict=None, named_tuple_type_name='NamedTuple', sep='/') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Convert a parametrized name between its string, tuple and dict forms. Built from a string template with `{field}` placeholders (and optional regexes for the fields). See `__init__` for the parameters and an example. #### extract(field, s) Extract a single item from an name * **Parameters:** * **field** – field of the item to extract * **s** – the string from which to extract it * **Returns:** the value for name #### info_dict(s) Get a dict with the arguments of an name (for example group, user, subuser, etc.) * **Parameters:** **s** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) * **Returns:** a dict holding the argument fields and values #### is_valid(s) Check if the name has the “upload format” (i.e. the kind of fields that are \_ids of fv_mgc, and what name means in most of the iatis system. * **Parameters:** **s** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the string to check * **Returns:** True iff name has the upload format #### replace_name_elements(s, \*\*elements_kwargs) Replace specific name argument values with others * **Parameters:** * **s** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the string to replace * **elements_kwargs** – the arguments to replace (and their values) * **Returns:** a new name #### str_to_dict(s) Get a dict with the arguments of an name (for example group, user, subuser, etc.) * **Parameters:** **s** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) * **Returns:** a dict holding the argument fields and values #### super_dict_to_str(d) Like dict_to_str, but the input dict can have extra keys that are not used by dict_to_str ### *class* dol.naming.StrTupleDictWithPrefix(template, format_dict=None, process_kwargs=None, process_info_dict=None, named_tuple_type_name='NamedTuple', sep='/') Bases: [`StrTupleDict`](_autosummary/dol.naming.html.md#dol.naming.StrTupleDict) Converting from and to strings, tuples, and dicts, but with partial “prefix” specs allowed. * **Parameters:** * **template** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) | [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)) – The string format template * **format_dict** – A {field_name: field_value_format_regex, …} dict * **process_kwargs** – A function taking the field=value pairs and producing a dict of processed {field: value,…} dict (where both fields and values could have been processed. This is useful when we need to process (format, default, etc.) fields, or their values, according to the other fields of values in the collection. A specification of {field: function_to_process_this_value,…} wouldn’t allow the full powers we are allowing here. * **process_info_dict** – A sort of converse of format_dict. This is a {field_name: field_conversion_func, …} dict that is used to convert info_dict values before returning them. * **name_separator** – Used ```pycon >>> ln = StrTupleDictWithPrefix('/home/{user}/fav/{num}.txt', ... format_dict={'user': '[^/]+', 'num': r'\d+'}, ... process_info_dict={'num': int}, ... sep='/' ... ) >>> ln.mk('USER', num=123) # making a string (with args or kwargs) '/home/USER/fav/123.txt' >>> ####### prefix methods ####### >>> ln.is_valid_prefix('/home/USER/fav/') True >>> ln.is_valid_prefix('/home/USER/fav/12') # False because too long False >>> ln.is_valid_prefix('/home/USER/fav') # False because too short False >>> ln.is_valid_prefix('/home/') # True because just right True >>> ln.is_valid_prefix('/home/USER/fav/123.txt') # full path, so output same as is_valid() method True >>> >>> ln.mk_prefix('ME') '/home/ME/fav/' >>> ln.mk_prefix(user='YOU', num=456) # full specification, so output same as same as mk() method '/home/YOU/fav/456.txt' ``` #### is_valid_prefix(s) Check if name is a valid prefix. * **Parameters:** **s** – a string (that might or might not be a valid prefix) * **Returns:** True iff name is a valid prefix ### dol.naming.dict_to_namedtuple(d, namedtuple_obj=None) ```pycon >>> from collections import namedtuple >>> NT = namedtuple('MyTuple', ('foo', 'hello')) >>> nt = NT(1, 42) >>> nt MyTuple(foo=1, hello=42) >>> d = namedtuple_to_dict(nt) >>> d {'foo': 1, 'hello': 42} >>> dict_to_namedtuple(d) NamedTupleFromDict(foo=1, hello=42) >>> dict_to_namedtuple(d, nt) MyTuple(foo=1, hello=42) ``` ### dol.naming.get_fields_from_template(template) Get list from {item} items of template string * **Parameters:** **template** – a “template” string (a string with {item} items – the kind that is used to mark token for str.format) * **Returns:** a list of the token items of the string, in the order they appear ```pycon >>> get_fields_from_template('this{is}an{example}of{a}template') ['is', 'example', 'a'] ``` ### dol.naming.mk_kwargs_trans(\*\*trans_func_for_key) Make a dict transformer from functions that depends solely on keys (of the dict to be transformed) Used to easily make process_kwargs and process_info_dict arguments for LinearNaming. ### dol.naming.mk_pattern_from_template_and_format_dict(template, format_dict=None, sep='/') Make a compiled regex to match template * **Parameters:** * **template** – A format string * **format_dict** – A dict whose keys are template fields and values are regex strings to capture them * **Returns:** a compiled regex Assert on *behavior* (matching) rather than the exact pattern string, so the examples hold on every OS (the field separator – and therefore the default capture class – is `/` on POSIX and `\` on Windows): ```pycon >>> p = mk_pattern_from_template_and_format_dict('{here}/and/{there}') >>> type(p) >>> p.match('HERE/and/1234').groupdict() {'here': 'HERE', 'there': '1234'} >>> p = mk_pattern_from_template_and_format_dict('{here}/and/{there}', {'there': r'\d+'}) >>> p.match('HERE/and/1234').groupdict() {'here': 'HERE', 'there': '1234'} >>> p.match('HERE/and/not_digits') is None # 'there' must be digits True ``` ### dol.naming.mk_store_from_path_format_store_cls(store=None, \*, subpath='', store_cls_kwargs=None, key_type=, keymap=, keymap_kwargs=None, name=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Wrap a store (instance or class) that uses string keys to make it into a store that uses a specific key format. * **Parameters:** * **store** – The instance or class to wrap * **subpath** – The subpath (defining the subset of the data pointed at by the URI * **store_cls_kwargs** – # if store is a class, the kwargs that you would have given the store_cls to make itself * **key_type** – The key type you want to interface with: `dict`, `tuple`, `namedtuple`, `str`, or one of those names as a string * **keymap** – # the keymap instance or class you want to use to map keys * **keymap_kwargs** – # if keymap is a cls, the kwargs to give it (besides the subpath) * **name** – The name to give the class the function will make here * **Returns:** An instance of a wrapped class ### Example ```python # Get a (session, bt) indexed LocalJsonStore s = mk_store_from_path_format_store_cls(LocalJsonStore, os.path.join(root_dir, 'd'), subpath='{session}/d/{bt}', keymap_kwargs=dict(process_info_dict={'session': int, 'bt': int})) ``` ### dol.naming.mk_tupled_store_from_path_format_store_cls(store=None, \*, subpath='', store_cls_kwargs=None, key_type=, keymap=, keymap_kwargs=None, name=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Wrap a store (instance or class) that uses string keys to make it into a store that uses a specific key format. * **Parameters:** * **store** – The instance or class to wrap * **subpath** – The subpath (defining the subset of the data pointed at by the URI * **store_cls_kwargs** – # if store is a class, the kwargs that you would have given the store_cls to make itself * **key_type** – The key type you want to interface with: `dict`, `tuple`, `namedtuple`, `str`, or one of those names as a string * **keymap** – # the keymap instance or class you want to use to map keys * **keymap_kwargs** – # if keymap is a cls, the kwargs to give it (besides the subpath) * **name** – The name to give the class the function will make here * **Returns:** An instance of a wrapped class ### Example ```python # Get a (session, bt) indexed LocalJsonStore s = mk_store_from_path_format_store_cls(LocalJsonStore, os.path.join(root_dir, 'd'), subpath='{session}/d/{bt}', keymap_kwargs=dict(process_info_dict={'session': int, 'bt': int})) ``` ### dol.naming.namedtuple_to_dict(nt) ```pycon >>> from collections import namedtuple >>> NT = namedtuple('MyTuple', ('foo', 'hello')) >>> nt = NT(1, 42) >>> nt MyTuple(foo=1, hello=42) >>> d = namedtuple_to_dict(nt) >>> d {'foo': 1, 'hello': 42} ``` ### dol.naming.template_to_pattern(mapping_dict, template) Weave a `{field}` template into a regex, substituting each field with its capture pattern and **regex-escaping the literal text between fields**. Escaping the literals is what makes this OS-independent: a template that is (or contains) a real filesystem path has backslashes on Windows (`C:\Users\...`), which are regex metacharacters – compiling them unescaped raises `re.error: incomplete escape \U`. Escaping also makes a literal `.` match a literal dot rather than any character. (This mirrors `KeyTemplate._compile_regex` and is why the result is never routed through `safe_compile`, which would re.escape the *whole* pattern on Windows and corrupt the capture groups.) ### dol.naming.update_fields_of_namedtuple(nt, , name_of_output_type=None, remove_fields=(), \*\*kwargs) Replace fields of namedtuple ```pycon >>> from collections import namedtuple >>> NT = namedtuple('NT', ('a', 'b', 'c')) >>> nt = NT(1,2,3) >>> nt NT(a=1, b=2, c=3) >>> update_fields_of_namedtuple(nt, c=3000) # replacing a single field NT(a=1, b=2, c=3000) >>> update_fields_of_namedtuple(nt, c=3000, a=1000) # replacing two fields NT(a=1000, b=2, c=3000) >>> update_fields_of_namedtuple(nt, a=1000, c=3000) # see that the original order doesn't change NT(a=1000, b=2, c=3000) >>> update_fields_of_namedtuple(nt, b=2000, d='hello') # replacing one field and adding a new one UpdatedNT(a=1, b=2000, c=3, d='hello') >>> # Now let's try controlling the name of the output type, remove fields, and add new ones >>> update_fields_of_namedtuple(nt, name_of_output_type='NewGuy', remove_fields=('a', 'c'), hello='world') NewGuy(b=2, hello='world') ``` ### dol.naming.validate_kwargs(kwargs_to_validate, validation_dict, validation_funs=None, all_kwargs_should_be_in_validation_dict=False, ignore_misunderstood_validation_instructions=False) Utility to validate a dict. It’s main use is to validate function arguments (expressing the validation checks in validation_dict) by doing validate_kwargs(locals()), usually in the beginning of the function (to avoid having more accumulated variables than we need in locals()) * **Parameters:** * **kwargs_to_validate** – as the name implies… * **validation_dict** – A dict specifying what to validate. Keys are usually name of variables (when feeding locals()) and values are dicts, themselves specifying check:check_val pairs where check is a string that points to a function (see validation_funs argument) and check_val is an object that the kwargs_to_validate value will be checked against. * **validation_funs** – A dict of check:check_function(val, check_val) where check_function is a function returning True if val is valid (with respect to check_val). * **all_kwargs_should_be_in_validation_dict** – If True, will raise an error if kwargs_to_validate contains keys that are not in validation_dict. * **ignore_misunderstood_validation_instructions** – If True, will raise an error if validation_dict contains a key that is not in validation_funs (safer, since if you mistype a key in validation_dict, the function will tell you so! * **Returns:** True if all the validations passed. ```pycon >>> validation_dict = { ... 'system': { ... 'be in': {'darwin', 'linux'} ... }, ... 'fv_version': { ... 'be a': int, ... 'be at least': 5 ... } ... } >>> validate_kwargs({'system': 'darwin'}, validation_dict) True >>> try: ... validate_kwargs({'system': 'windows'}, validation_dict) ... except AssertionError as e: ... assert str(e).startswith('system must be in') # omitting the set because inconsistent order >>> try: ... validate_kwargs({'fv_version': 9.9}, validation_dict) ... except AssertionError as e: ... print(e) fv_version must be a >>> try: ... validate_kwargs({'fv_version': 4}, validation_dict) ... except AssertionError as e: ... print(e) fv_version must be at least 5 >>> validate_kwargs({'fv_version': 6}, validation_dict) True ``` # _autosummary/dol.paths.html.md # dol.paths Module for path (and path-like) object manipulation ### Examples ```pycon >>> d = {'a': {'b': {'c': 1, 'd': 2}, 'e': 3}} >>> list(path_filter(lambda p, k, v: v == 2, d)) [('a', 'b', 'd')] >>> path_get(d, ('a', 'b', 'd')) 2 >>> path_set(d, ('a', 'b', 'd'), 4) >>> d {'a': {'b': {'c': 1, 'd': 4}, 'e': 3}} >>> path_set(d, ('a', 'b', 'new_ab_key'), 42) >>> d {'a': {'b': {'c': 1, 'd': 4, 'new_ab_key': 42}, 'e': 3}} ``` ### Functions | [`add_prefix_filtering`](_autosummary/dol.paths.html.md#dol.paths.add_prefix_filtering)([store, ...]) | Make a missing key that is a prefix of existing keys return the sub-mapping of those keys (so `s['a/']` lists everything "under" `a/`). | |-------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| | [`cast_to_int_if_numeric_str`](_autosummary/dol.paths.html.md#dol.paths.cast_to_int_if_numeric_str)(k) | Cast `k` to `int` if it is a numeric string; return it unchanged otherwise. | | [`chain_of_getters`](_autosummary/dol.paths.html.md#dol.paths.chain_of_getters)(getters[, obj, k, ...]) | If `k` is a string, tries to get `k` as an attribute of `obj` first, and if that fails, gets it as `obj[k]` | | [`ensure_path_extender_func`](_autosummary/dol.paths.html.md#dol.paths.ensure_path_extender_func)(path_extender) | Ensure that the path_extender is a function that takes a path and a key and returns a new path. | | [`flatten_dict`](_autosummary/dol.paths.html.md#dol.paths.flatten_dict)(d[, sep, parent_path, ...]) | Flatten a nested dictionary into a flat one, using key-paths as keys. | | [`flattened_dict_items`](_autosummary/dol.paths.html.md#dol.paths.flattened_dict_items)(d[, sep, parent_path, ...]) | Yield flattened key-value pairs from a nested dictionary. | | [`get_attr_or_item`](_autosummary/dol.paths.html.md#dol.paths.get_attr_or_item)(obj, k) | If `k` is a string, tries to get `k` as an attribute of `obj` first, and if that fails, gets it as `obj[k]` | | [`getitem`](_autosummary/dol.paths.html.md#dol.paths.getitem)(obj, k) | Return `obj[k]`. | | [`handle_prefixes`](_autosummary/dol.paths.html.md#dol.paths.handle_prefixes)([store, prefix, ...]) | A store decorator that handles prefixes. | | [`identity`](_autosummary/dol.paths.html.md#dol.paths.identity)(x) | Return `x`. | | [`keys_and_indices_path`](_autosummary/dol.paths.html.md#dol.paths.keys_and_indices_path)(str_path, \*[, sep, ...]) | Transforms a string path separated by a specified separator into a tuple of keys and indices. | | [`leaf_paths`](_autosummary/dol.paths.html.md#dol.paths.leaf_paths)(d[, sep, parent_path, egress]) | Get a dictionary of leaf paths of a nested dictionary. | | [`mk_relative_path_store`](_autosummary/dol.paths.html.md#dol.paths.mk_relative_path_store)([store_cls, name, ...]) | | | [`path_edit`](_autosummary/dol.paths.html.md#dol.paths.path_edit)(d[, edits]) | Make a series of (in place) edits to a Mapping, specifying `(path, value)` pairs. | | [`path_filter`](_autosummary/dol.paths.html.md#dol.paths.path_filter)(pkv_filt, d, \*[, leafs_only, ...]) | Walk a dict, yielding paths to values that pass the `pkv_filt` | | [`path_get`](_autosummary/dol.paths.html.md#dol.paths.path_get)(obj, path[, on_error, sep, ...]) | Get elements of a mapping through a path to be called recursively. | | [`paths_getter`](_autosummary/dol.paths.html.md#dol.paths.paths_getter)(paths[, obj, egress, on_error, ...]) | Returns (path, values) pairs of the given paths in the given object. | | [`prefixless_view`](_autosummary/dol.paths.html.md#dol.paths.prefixless_view)([store, prefix, \_\_module_\_, ...]) | Wrap `store` so that keys are seen without `prefix` (added back on access). | | [`raise_on_error`](_autosummary/dol.paths.html.md#dol.paths.raise_on_error)(d) | `on_error` policy for `path_get`: re-raise the caught error. | | [`rel_path_wrap`](_autosummary/dol.paths.html.md#dol.paths.rel_path_wrap)(o, \_prefix) | | | [`return_empty_tuple_on_error`](_autosummary/dol.paths.html.md#dol.paths.return_empty_tuple_on_error)(d) | `on_error` policy for `path_get`: return `()`. | | [`return_none_on_error`](_autosummary/dol.paths.html.md#dol.paths.return_none_on_error)(d) | `on_error` policy for `path_get`: return `None`. | | [`search_paths`](_autosummary/dol.paths.html.md#dol.paths.search_paths)(d, pkv_filt, \*[, leafs_only, ...]) | backwards compatibility quasi-alias (arguments are flipped) Use path_filter instead, since search_paths will be deprecated. | | [`separate_keys_with_separator`](_autosummary/dol.paths.html.md#dol.paths.separate_keys_with_separator)(obj[, sep]) | Split a string path on `sep` and cast numeric parts to `int`; a non-string iterable is only cast element-wise. | | [`separator_based_path_extender`](_autosummary/dol.paths.html.md#dol.paths.separator_based_path_extender)(path, key, sep) | Extends a given path with a new key using the specified separator. | | [`split_if_str`](_autosummary/dol.paths.html.md#dol.paths.split_if_str)(obj[, sep]) | Split `obj` on `sep` if it is a string; return it unchanged otherwise. | | [`str_template_key_trans`](_autosummary/dol.paths.html.md#dol.paths.str_template_key_trans)(template, key_type[, ...]) | Make a key trans object that translates from a string \_id to a dict, tuple, or namedtuple key (and back) | | [`string_unparse`](_autosummary/dol.paths.html.md#dol.paths.string_unparse)(parsing_result) | The inverse of string.Formatter.parse | ### Classes | [`ExplicitKeysWithPrefixRelativization`](_autosummary/dol.paths.html.md#dol.paths.ExplicitKeysWithPrefixRelativization)(...[, ...]) | dol.base.Keys implementation that gets it's keys explicitly from a collection given at initialization time. | |-----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| | [`KeyPath`](_autosummary/dol.paths.html.md#dol.paths.KeyPath)([path_sep, \_path_type, ...]) | A key mapper that converts from an iterable key (default tuple) to a string (given a path-separator str) | | [`KeyTemplate`](_autosummary/dol.paths.html.md#dol.paths.KeyTemplate)(template, \*[, field_patterns, ...]) | A class for parsing and generating keys based on a template. | | [`PathKeyTypes`](_autosummary/dol.paths.html.md#dol.paths.PathKeyTypes)(\*values) | Enum of the path key forms: `str`, `dict`, `tuple`, `namedtuple`. | | [`PathMappedData`](_autosummary/dol.paths.html.md#dol.paths.PathMappedData)(src, key_collection[, ...]) | A collection of keys with a key_to_value function to lazy load values. | | [`PrefixRelativization`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativization)([_prefix]) | A key wrap that allows one to interface with absolute paths through relative paths. | | [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin)() | Mixin that adds a intercepts the \_id_of_key an \_key_of_id methods, transforming absolute keys to relative ones. | | [`RelativePathKeyMapper`](_autosummary/dol.paths.html.md#dol.paths.RelativePathKeyMapper)(prefix) | Key mapper adding `prefix` on the way in and removing it on the way out. | ### *class* dol.paths.ExplicitKeysWithPrefixRelativization(key_collection, \_prefix=None) Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) dol.base.Keys implementation that gets it’s keys explicitly from a collection given at initialization time. The key_collection must be a collections.abc.Collection (such as list, tuple, set, etc.) ```pycon >>> from dol.base import Store >>> s = ExplicitKeysWithPrefixRelativization(key_collection=['/root/of/foo', '/root/of/bar', '/root/for/alice']) >>> keys = Store(store=s) >>> 'of/foo' in keys True >>> 'not there' in keys False >>> list(keys) ['of/foo', 'of/bar', 'for/alice'] ``` ### *class* dol.paths.KeyPath(path_sep='/', \_path_type=, \*, create_missing=False, mk_missing=None, explore_further=None, may_create=None, on_create=, max_created=None, max_levels=20, verify_writeback=False, writeback_lock=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A key mapper that converts from an iterable key (default tuple) to a string (given a path-separator str) * **Parameters:** * **path_sep** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The path separator (used to make string paths from iterable paths and visa versa * **\_path_type** ([`type`](https://docs.python.org/3/builtins/functions.html#type) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – The type of the outcoming (inner) path. But really, any function to * **to** (*convert from a list*) – the outer path type we want. With `'/'` as a separator: ```pycon >>> kp = KeyPath(path_sep='/') >>> kp._key_of_id(('a', 'b', 'c')) 'a/b/c' >>> kp._id_of_key('a/b/c') ('a', 'b', 'c') ``` With `'.'` as a separator: ```pycon >>> kp = KeyPath(path_sep='.') >>> kp._key_of_id(('a', 'b', 'c')) 'a.b.c' >>> kp._id_of_key('a.b.c') ('a', 'b', 'c') >>> kp = KeyPath(path_sep=':::', _path_type=dict.fromkeys) >>> _id = dict.fromkeys('abc') >>> _id {'a': None, 'b': None, 'c': None} >>> kp._key_of_id(_id) 'a:::b:::c' >>> kp._id_of_key('a:::b:::c') {'a': None, 'b': None, 'c': None} ``` Calling a `KeyPath` instance on a store wraps it so we can have path access to it. ```pycon >>> s = {'a': {'b': {'c': 42}}} >>> s['a']['b']['c'] 42 >>> # Now let's wrap the store >>> s = KeyPath('.')(s) >>> s['a.b.c'] 42 >>> s['a.b.c'] = 3.14 >>> s['a.b.c'] 3.14 >>> del s['a.b.c'] >>> s {'a': {'b': {}}} ``` #### NOTE By default `KeyPath` reads with paths only when all the keys of the path are valid (i.e. have a value), and, just like a `dict`, will *not* create intermediate nested values for you on write. Pass `create_missing=True` to opt into write-through autovivification: missing intermediates are created on write (like `collections.defaultdict`, but with an optional contextual per-level `mk_missing(ctx)` factory), and the change persists correctly even through persistent / copy-semantics stores. See `misc/docs/dol_issue16_design.md`. ```pycon >>> s = KeyPath('.', create_missing=True)({}) >>> s['a.b.c'] = 42 >>> s['a.b.c'] 42 ``` #### on_create() Default `on_create` hook: announce a fabricated intermediate. A structurally-valid *typo* on write would otherwise silently create a bogus branch; warning keeps opted-in creation from being silent. Pass `on_create=None` to silence (e.g. bulk tree building). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### *class* dol.paths.KeyTemplate(template, \*, field_patterns=None, to_str_funcs=None, from_str_funcs=None, simple_str_sep=', ', namedtuple_type_name='NamedTuple', dflt_pattern='.\*', dflt_field_name=, normalize_paths=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A class for parsing and generating keys based on a template. * **Parameters:** * **template** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A template string with fields to be extracted or filled in. * **field_patterns** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – A dictionary of field names and their regex patterns. * **simple_str_sep** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A separator string for simple strings (i.e. strings without fields). * **namedtuple_type_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the namedtuple type to use for namedtuple fields. * **dflt_pattern** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The default pattern to use for fields that don’t have a pattern specified. * **to_str_funcs** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – A dictionary of field names and their functions to convert them to strings. * **from_str_funcs** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – A dictionary of field names and their functions to convert them from strings. ### Examples ```pycon >>> st = KeyTemplate( ... 'root/{name}/v_{version}.json', ... field_patterns={'version': r'\d+'}, ... from_str_funcs={'version': int}, ... ) ``` And now you have a template that can be used to convert between various representations of the template: You can extract fields from strings, generate strings from fields, etc. ```pycon >>> st.str_to_dict("root/dol/v_9.json") {'name': 'dol', 'version': 9} >>> st.dict_to_str({'name': 'meshed', 'version': 42}) 'root/meshed/v_42.json' >>> st.dict_to_tuple({'name': 'meshed', 'version': 42}) ('meshed', 42) >>> st.tuple_to_dict(('i2', 96)) {'name': 'i2', 'version': 96} >>> st.str_to_tuple("root/dol/v_9.json") ('dol', 9) >>> st.tuple_to_str(('front', 11)) 'root/front/v_11.json' >>> st.str_to_namedtuple("root/dol/v_9.json") NamedTuple(name='dol', version=9) >>> st.str_to_simple_str("root/dol/v_9.json") 'dol,9' >>> st_clone = st.clone(simple_str_sep='/') >>> st_clone.str_to_simple_str("root/dol/v_9.json") 'dol/9' ``` With `st.key_codec`, you can make a `KeyCodec` for the given source (decoded) and target (encoded) types. A `key_codec` is a codec; it has an encoder and a decoder. ```pycon >>> key_codec = st.key_codec('tuple', 'str') >>> encoder, decoder = key_codec >>> decoder('root/dol/v_9.json') ('dol', 9) >>> encoder(('dol', 9)) 'root/dol/v_9.json' ``` If you have a `Mapping`, you can use `key_codec` as a decorator to wrap the mapping with a key mappings. ```pycon >>> store = { ... 'root/meshed/v_151.json': '{"downloads": 41, "type": "productivity"}', ... 'root/dol/v_9.json': '{"downloads": 132, "type": "utility"}', ... } >>> >>> accessor = key_codec(store) >>> list(accessor) [('meshed', 151), ('dol', 9)] >>> accessor['i2', 4] = '{"downloads": 274, "type": "utility"}' >>> list(store) ['root/meshed/v_151.json', 'root/dol/v_9.json', 'root/i2/v_4.json'] >>> store['root/i2/v_4.json'] '{"downloads": 274, "type": "utility"}' ``` #### NOTE If your store contains keys that don’t fit the format, key_codec will raise a `ValueError`. To remedy this, you can use the `st.filt_iter` to filter out keys that don’t fit the format, before you wrap the store with `st.key_codec`. ```pycon >>> store = { ... 'root/meshed/v_151.json': '{"downloads": 41, "type": "productivity"}', ... 'root/dol/v_9.json': '{"downloads": 132, "type": "utility"}', ... 'root/not/the/right/format': "something else" ... } >>> accessor = st.filt_iter('str')(store) >>> list(accessor) ['root/meshed/v_151.json', 'root/dol/v_9.json'] >>> accessor = st.key_codec('tuple', 'str')(st.filt_iter('str')(store)) >>> list(accessor) [('meshed', 151), ('dol', 9)] >>> accessor['dol', 9] '{"downloads": 132, "type": "utility"}' ``` You can also ask any (handled) combination of field types: ```pycon >>> key_codec = st.key_codec('tuple', 'dict') >>> key_codec.encoder(('i2', 96)) {'name': 'i2', 'version': 96} >>> key_codec.decoder({'name': 'fantastic', 'version': 4}) ('fantastic', 4) ``` #### dict_to_namedtuple(params) Generates a namedtuple from the dictionary values based on the template. ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> App = st.dict_to_namedtuple({'i01_': 'life', 'ver': 42}) >>> App NamedTuple(i01_='life', ver=42) ``` #### dict_to_str(params) Generates a string from the dictionary values based on the template. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.dict_to_str({'i01_': 'life', 'ver': 42}) 'root/life/v_042.json' ``` #### dict_to_tuple(params) Generates a tuple from the dictionary values based on the template. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.str_to_tuple('root/life/v_42.json') ('life', 42) ``` #### filt_iter(field_type='str') Makes a store decorator that filters out keys that don’t match the template given field type. ```pycon >>> store = { ... 'root/meshed/v_151.json': '{"downloads": 41, "type": "productivity"}', ... 'root/dol/v_9.json': '{"downloads": 132, "type": "utility"}', ... 'root/not/the/right/format': "something else" ... } >>> filt = KeyTemplate('root/{pkg}/v_{version}.json') >>> filtered_store = filt.filt_iter('str')(store) >>> list(filtered_store) ['root/meshed/v_151.json', 'root/dol/v_9.json'] ``` #### key_codec(decoded='tuple', encoded='str') Makes a `KeyCodec` for the given source and target types. ```pycon >>> st = KeyTemplate( ... 'root/{name}/v_{version}.json', ... field_patterns={'version': r'\d+'}, ... from_str_funcs={'version': int}, ... ) ``` A `key_codec` is a codec; it has an encoder and a decoder. ```pycon >>> key_codec = st.key_codec('tuple', 'str') >>> encoder, decoder = key_codec >>> decoder('root/dol/v_9.json') ('dol', 9) >>> encoder(('dol', 9)) 'root/dol/v_9.json' ``` If you have a `Mapping`, you can use `key_codec` as a decorator to wrap the mapping with a key mappings. ```pycon >>> store = { ... 'root/meshed/v_151.json': '{"downloads": 41, "type": "productivity"}', ... 'root/dol/v_9.json': '{"downloads": 132, "type": "utility"}', ... } >>> >>> accessor = key_codec(store) >>> list(accessor) [('meshed', 151), ('dol', 9)] >>> accessor['i2', 4] = '{"downloads": 274, "type": "utility"}' >>> list(store) ['root/meshed/v_151.json', 'root/dol/v_9.json', 'root/i2/v_4.json'] >>> store['root/i2/v_4.json'] '{"downloads": 274, "type": "utility"}' ``` #### NOTE If your store contains keys that don’t fit the format, key_codec will raise a `ValueError`. To remedy this, you can use the `st.filt_iter` to filter out keys that don’t fit the format, before you wrap the store with `st.key_codec`. #### match_str(s) Returns True iff the string matches the template. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.match_str('root/life/v_042.json') True >>> st.match_str('this/does/not_match') False ``` #### namedtuple_to_dict(nt) Converts a namedtuple to a dictionary. ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> App = st.dict_to_namedtuple({'i01_': 'life', 'ver': 42}) >>> st.namedtuple_to_dict(App) {'i01_': 'life', 'ver': 42} ``` #### simple_str_to_str(ss) Converts a simple character-delimited string to a string. ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... simple_str_sep='-', ... ) >>> st.simple_str_to_str('life-042') 'root/life/v_042.json' ``` #### simple_str_to_tuple(ss) Converts a simple character-delimited string to a dict. ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... simple_str_sep='-', ... ) >>> st.simple_str_to_tuple('life-042') ('life', 42) ``` #### single_to_str(k) Generates a string from the single value based on the template. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> st = KeyTemplate( ... r'root/life/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.single_to_str(42) 'root/life/v_042.json' ``` #### str_to_dict(s) Parses the input string and returns a dictionary of extracted values. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', ... from_str_funcs={'ver': int}, ... ) >>> st.str_to_dict('root/life/v_30.json') {'i01_': 'life', 'ver': 30} ``` #### str_to_namedtuple(s) Converts a string to a namedtuple. ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> App = st.str_to_namedtuple('root/life/v_042.json') >>> App NamedTuple(i01_='life', ver=42) ``` #### str_to_simple_str(s) Converts a string to a simple string (i.e. a simple character-delimited string). ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.str_to_simple_str('root/life/v_042.json') 'life,042' >>> st_clone = st.clone(simple_str_sep='-') >>> st_clone.str_to_simple_str('root/life/v_042.json') 'life-042' ``` #### str_to_single(s) Parses the input string and returns a single value. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ```pycon >>> st = KeyTemplate( ... r'root/life/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.str_to_single('root/life/v_42.json') 42 ``` #### str_to_tuple(s) Parses the input string and returns a tuple of extracted values. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.str_to_tuple('root/life/v_42.json') ('life', 42) ``` #### tuple_to_dict(param_vals) Generates a dictionary from the tuple values based on the template. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.tuple_to_dict(('life', 42)) {'i01_': 'life', 'ver': 42} ``` #### tuple_to_str(param_vals) Generates a string from the tuple values based on the template. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> st = KeyTemplate( ... r'root/{}/v_{ver:03.0f:\d+}.json', from_str_funcs={'ver': int}, ... ) >>> st.tuple_to_str(('life', 42)) 'root/life/v_042.json' ``` ### *class* dol.paths.PathKeyTypes(\*values) Bases: [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) Enum of the path key forms: `str`, `dict`, `tuple`, `namedtuple`. ### *class* dol.paths.PathMappedData(src, key_collection, getter=, \*, key_to_value=None) Bases: [`KeysReader`](_autosummary/dol.explicit.html.md#dol.explicit.KeysReader) A collection of keys with a key_to_value function to lazy load values. `PathMappedData` is particularly useful in cases where you want to have a mapping that lazy-loads values for keys from an explicit collection. Keywords: Lazy-evaluation, Mapping * **Parameters:** * **data** – The mapping to extract data from * **paths** – The paths to extract data from the mapping ### Example ```pycon >>> data = { ... 'a': { ... 'b': [{'c': 1}, {'c': 2}], ... 'd': 'bar' ... } ... } >>> paths = ['a.d', 'a.b.0.c'] >>> >>> d = PathMappedData(data, paths) >>> list(d) ['a.d', 'a.b.0.c'] >>> d['a.d'] 'bar' >>> d['a.b.0.c'] 1 ``` Now, data does contain a key path for ‘a.b.1.c’: ```pycon >>> d.getter(d.src, 'a.b.1.c') 2 ``` But since we didn’t mention it in our paths parameter, it will raise a KeyError if we try to access it via the `PathMappedData` object: ```pycon >>> d['a.b.1.c'] Traceback (most recent call last): ... KeyError: "Key a.b.1.c was not found....key_collection attribute)" ``` ### *class* dol.paths.PrefixRelativization(\_prefix='') Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin) A key wrap that allows one to interface with absolute paths through relative paths. The original intent was for local files. Instead of referencing files through an absolute path such as: > */A/VERY/LONG/ROOT/FOLDER/the/file/we.want* we can instead reference the file as: > *the/file/we.want* But PrefixRelativization can be used, not only for local paths, but when ever a string reference is involved. In fact, not only strings, but any key object that has a \_\_len_\_, \_\_add_\_, and subscripting. ### *class* dol.paths.PrefixRelativizationMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mixin that adds a intercepts the \_id_of_key an \_key_of_id methods, transforming absolute keys to relative ones. Designed to work with string keys, where absolute and relative are relative to a \_prefix attribute (assumed to exist). The cannonical use case is when keys are absolute file paths, but we want to identify data through relative paths. Instead of referencing files through an absolute path such as `/A/VERY/LONG/ROOT/FOLDER/the/file/we.want` we can instead reference the file as `the/file/we.want`. Note though, that PrefixRelativizationMixin can be used, not only for local paths, but when ever a string reference is involved. In fact, not only strings, but any key object that has a \_\_len_\_, \_\_add_\_, and subscripting. When subclassed, should be placed before the class defining \_id_of_key an \_key_of_id. Also, assumes that a (string) \_prefix attribute will be available. ```pycon >>> from dol.base import Store >>> from collections import UserDict >>> >>> class MyStore(PrefixRelativizationMixin, Store): ... def __init__(self, store, _prefix='/root/of/data/'): ... super().__init__(store) ... self._prefix = _prefix ... >>> s = MyStore(store=dict()) # using a dict as our store >>> s['foo'] = 'bar' >>> assert s['foo'] == 'bar' >>> s['too'] = 'much' >>> assert list(s.keys()) == ['foo', 'too'] >>> # Everything looks normal, but are the actual keys behind the hood? >>> s._id_of_key('foo') '/root/of/data/foo' >>> # see when iterating over s.items(), we get the interface view: >>> list(s.items()) [('foo', 'bar'), ('too', 'much')] >>> # but if we ask the store we're actually delegating the storing to, we see what the keys actually are. >>> s.store.items() dict_items([('/root/of/data/foo', 'bar'), ('/root/of/data/too', 'much')]) ``` ### *class* dol.paths.RelativePathKeyMapper(prefix) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Key mapper adding `prefix` on the way in and removing it on the way out. ### dol.paths.add_prefix_filtering(store=None, , relativize_prefix=False, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make a missing key that is a prefix of existing keys return the sub-mapping of those keys (so `s['a/']` lists everything “under” `a/`). ```pycon >>> d = {'a/b': 1, 'a/c': 2, 'd/e': 3, 'f': 4} >>> s = add_prefix_filtering(d) >>> assert s['a/'] == {'a/b': 1, 'a/c': 2} ``` Demo usage on a `Mapping` type: ```pycon >>> from collections import UserDict >>> D = add_prefix_filtering(UserDict) >>> s = D(d) >>> assert s['a/'] == {'a/b': 1, 'a/c': 2} ``` ### dol.paths.cast_to_int_if_numeric_str(k) Cast `k` to `int` if it is a numeric string; return it unchanged otherwise. ### dol.paths.chain_of_getters(getters, obj=None, k=None, \*, caught_errors=(, )) If `k` is a string, tries to get `k` as an attribute of `obj` first, and if that fails, gets it as `obj[k]` ### dol.paths.ensure_path_extender_func(path_extender) Ensure that the path_extender is a function that takes a path and a key and returns a new path. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`)] ### dol.paths.flatten_dict(d, sep='.', \*, parent_path=None, visit_nested=>, egress=) Flatten a nested dictionary into a flat one, using key-paths as keys. See also `leaf_paths` for a related function that returns paths to leaf values. * **Parameters:** * **d** – The dictionary to flatten * **sep** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`)]]) – The separator to use for joining keys, or a function that takes a path and a key and returns a new path. * **parent_path** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`)]) – The path to the parent of the current dict * **visit_nested** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – A function that returns True if a value should be visited * **egress** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Generator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Generator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`None`](https://docs.python.org/3/builtins/constants.html#None), [`None`](https://docs.python.org/3/builtins/constants.html#None)]], [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – A function that takes a generator of key-value pairs and returns a mapping ```pycon >>> d = {'a': {'b': 2}, 'c': 3} >>> flatten_dict(d) {'a.b': 2, 'c': 3} >>> flatten_dict(d, sep='/') {'a/b': 2, 'c': 3} ``` ### dol.paths.flattened_dict_items(d, sep='.', \*, parent_path=None, visit_nested=>) Yield flattened key-value pairs from a nested dictionary. * **Return type:** [`Generator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Generator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`None`](https://docs.python.org/3/builtins/constants.html#None), [`None`](https://docs.python.org/3/builtins/constants.html#None)] ### dol.paths.get_attr_or_item(obj, k) If `k` is a string, tries to get `k` as an attribute of `obj` first, and if that fails, gets it as `obj[k]` #### WARNING The hardcoded priority choices of this function regarding when to try k as an item, index, or attribute, don’t apply to every case, so you may want to use an explicit value getter to be more robust! # >>> d = {‘a’: [1, {‘items’: 2, ‘3’: 33, 3: 42}]} ```pycon >>> get_attr_or_item({'items': 2}, 'items') 2 ``` But if “items” is not there as a key of the object, the attribute is found: ```pycon >>> get_attr_or_item({'not_items': 2}, 'items') ``` Both integers and string integers will work to get an item if obj is not a Mapping. ```pycon >>> get_attr_or_item([7, 21, 42], 2) 42 >>> get_attr_or_item([7, 21, 42], '2') 42 ``` If you’re dealling with a Mapping, you can get both integer and string keys, and if you have both types in your Mapping, you’ll get the right one! ```pycon >>> get_attr_or_item({2: 'numerical key', '2': 'string key'}, 2) 'numerical key' >>> get_attr_or_item({2: 'numerical key', '2': 'string key'}, '2') 'string key' ``` If you don’t have the numerical version, the string version will still find your numerical key. ```pycon >>> get_attr_or_item({2: 'numerical key'}, '2') 'numerical key' ``` The opposite is not true though: If you ask for an integer key, it will not find a string version of it. ```pycon >>> get_attr_or_item({'2': 'string key'}, 2) # +IGNORE_EXCEPTION_DETAIL Traceback (most recent call last): ... KeyError: 2 ``` ### dol.paths.getitem(obj, k) Return `obj[k]`. ### dol.paths.handle_prefixes(store=None, , prefix=None, filter_prefix=True, relativize_prefix=True, default_prefix='', \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) A store decorator that handles prefixes. If aggregates several prefix-related functionalities. It will (by default) - Filter the store so that only the keys starting with given prefix are accessible. - Relativize the keys (provide a view where the prefix is removed from the keys) * **Parameters:** * **store** – The store to wrap * **prefix** – The prefix to use. If None and the store is an instance (not type), will take the longest common prefix as the prefix. * **filter_prefix** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to filter out keys that don’t start with the prefix * **relativize_prefix** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to relativize the prefix * **default_prefix** – The default prefix to use if no prefix is given and the store is a type (not instance) ```pycon >>> d = {'/ROOT/of/every/thing': 42, '/ROOT/of/this/too': 0} >>> dd = handle_prefixes(d, prefix='/ROOT/of/') >>> dd['foo'] = 'bar' >>> dict(dd.items()) # gives us what you would expect {'every/thing': 42, 'this/too': 0, 'foo': 'bar'} >>> dict(dd.store) # but see where the underlying store actually wrote 'bar': {'/ROOT/of/every/thing': 42, '/ROOT/of/this/too': 0, '/ROOT/of/foo': 'bar'} ``` ### dol.paths.identity(x) Return `x`. ### dol.paths.keys_and_indices_path(str_path, , sep='.', index_pattern='\\\\[(\\\\d+)\\\\]') Transforms a string path separated by a specified separator into a tuple of keys and indices. Bracketed indices are extracted as integers. This function is meant to be used in as the key_transformer argument of path_get etc. * **Parameters:** * **path** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The input path string, e.g., “a21-59c.message[2].user”. * **sep** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The separator used to split the path, default is ‘.’. * **index_pattern** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The regular expression pattern to match bracketed indices * **Returns:** A tuple representation of the path, e.g., (“a21-59c”, “message”, 2, “user”). * **Return type:** [*tuple*](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### Example ```pycon >>> keys_and_indices_path("a21-59c.message[2].user") ('a21-59c', 'message', 2, 'user') ``` ### dol.paths.leaf_paths(d, sep='.', \*, parent_path=None, egress=) Get a dictionary of leaf paths of a nested dictionary. Given a nested dictionary, returns a similarly structured dictionary where each leaf value is replaced by its flattened path. The ‘sep’ parameter can be either a string or a callable. Original use case: You used flatten_dict to flatten a nested dictionary, referencing your values with paths, but maybe you’d like to know what the paths that your nested dictionary is going to flatten to are. This function does that. The output is a dict with the same keys and structure as the input, but the leaf values are replaced by the paths that would be used to access them in a flat dict. * **Parameters:** * **d** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), `Union`[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), `Union`[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`), NestedMapping[KT, VT]]]]]) – The nested dictionary to get the leaf paths from * **sep** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`)]]) – The separator to use for joining keys, or a function that takes a path and a key and returns a new path. * **parent_path** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`)]) – The path to the parent of the current dict * **egress** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Generator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Generator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`None`](https://docs.python.org/3/builtins/constants.html#None), [`None`](https://docs.python.org/3/builtins/constants.html#None)]], [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – A function that takes a generator of key-value pairs and returns a mapping * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), `Union`[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`)]] ### Example ```pycon >>> leaf_paths({'a': {'b': 2}, 'c': 3}) {'a': {'b': 'a.b'}, 'c': 'c'} ``` ```pycon >>> leaf_paths({'a': {'b': 2}, 'c': 3}, sep="/") {'a': {'b': 'a/b'}, 'c': 'c'} ``` ```pycon >>> leaf_paths({'a': {'b': 2}, 'c': 3}, sep=lambda p, k: f"{p}-{k}" if p else k) {'a': {'b': 'a-b'}, 'c': 'c'} ``` ### dol.paths.mk_relative_path_store(store_cls=None, , name=None, with_key_validation=False, prefix_attr='_prefix', \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) * **Parameters:** * **store_cls** – The base store to wrap (subclass) * **name** – The name of the new store (by default ‘RelPath’ + store_cls._\_name_\_) * **with_key_validation** – Whether keys should be validated upon access (store_cls must have an is_valid_key method * **Returns:** A new class that uses relative paths (i.e. where \_prefix is automatically added to incoming keys, and the len(_prefix) first characters are removed from outgoing keys. ```pycon >>> # The dynamic way (if you try this at home, be aware of the pitfalls of the dynamic way >>> # -- but don't just believe the static dogmas). >>> MyStore = mk_relative_path_store(dict) # wrap our favorite store: A dict. >>> s = MyStore() # make such a store >>> s._prefix = '/ROOT/' >>> s['foo'] = 'bar' >>> dict(s.items()) # gives us what you would expect {'foo': 'bar'} >>> # but under the hood, the dict we wrapped actually contains the '/ROOT/' prefix >>> dict(s.store) {'/ROOT/foo': 'bar'} >>> >>> # The static way: Make a class that will integrate the _prefix at construction time. >>> class MyStore(mk_relative_path_store(dict)): # Indeed, mk_relative_path_store(dict) is a class you can subclass ... def __init__(self, _prefix, *args, **kwargs): ... self._prefix = _prefix ``` You can choose the name you want that prefix to have as an attribute (we’ll still make a hidden ‘_prefix’ attribute for internal use, but at least you can have an attribute with the name you want. ```pycon >>> MyRelStore = mk_relative_path_store(dict, prefix_attr='rootdir') >>> s = MyRelStore() >>> s.rootdir = '/ROOT/' ``` ```pycon >>> s['foo'] = 'bar' >>> dict(s.items()) # gives us what you would expect {'foo': 'bar'} >>> # but under the hood, the dict we wrapped actually contains the '/ROOT/' prefix >>> dict(s.store) {'/ROOT/foo': 'bar'} ``` ### dol.paths.path_edit(d, edits=()) Make a series of (in place) edits to a Mapping, specifying `(path, value)` pairs. * **Parameters:** * **d** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – The mapping to edit. * **edits** (`Union`[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)]]]) – An iterable of `(path, value)` tuples, or `path: value` Mapping. * **Returns:** The edited mapping. * **Return type:** [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) ```pycon >>> d = {'a': 1} >>> path_edit(d, [(['b', 'c'], 2), ('d.e.f', 3)]) {'a': 1, 'b': {'c': 2}, 'd': {'e': {'f': 3}}} ``` Changes happened also inplace (so if you don’t want that, make a deepcopy first): ```pycon >>> d {'a': 1, 'b': {'c': 2}, 'd': {'e': {'f': 3}}} ``` You can also pass a dict of edits. ```pycon >>> path_edit(d, {'a': 4, 'd.e.f': 5}) {'a': 4, 'b': {'c': 2}, 'd': {'e': {'f': 5}}} ``` ### dol.paths.path_filter(pkv_filt, d, , leafs_only=True, breadth_first=False) Walk a dict, yielding paths to values that pass the `pkv_filt` * **Parameters:** * **pkv_filt** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – A function that takes a path, key, and value, and returns `True` if the path should be yielded, and `False` otherwise * **d** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – The `Mapping` to walk (scan through) * **leafs_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to yield only paths to leafs (default), or to yield paths to all values that pass the `pkv_filt`. * **breadth_first** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to perform breadth-first traversal (instead of the default depth-first traversal). * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`)] * **Returns:** An iterator of paths to values that pass the `pkv_filt` ### Example ```pycon >>> d = {'a': {'b': {'c': 1, 'd': 2}, 'e': 3}} >>> list(path_filter(lambda p, k, v: v == 2, d)) [('a', 'b', 'd')] ``` ```pycon >>> mm = { ... 'a': {'b': {'c': 42}}, ... 'aa': {'bb': {'cc': 'meaning of life'}}, ... 'aaa': {'bbb': 314}, ... } >>> return_path_if_int_leaf = lambda p, k, v: (p, v) if isinstance(v, int) else None >>> paths = list(path_filter(return_path_if_int_leaf, mm)) >>> paths # only the paths to the int leaves are returned [('a', 'b', 'c'), ('aaa', 'bbb')] ``` The `pkv_filt` argument can use path, key, and/or value to define your search query. For example, let’s extract all the paths that have depth at least 3. ```pycon >>> paths = list(path_filter(lambda p, k, v: len(p) >= 3, mm)) >>> paths [('a', 'b', 'c'), ('aa', 'bb', 'cc')] ``` The rationale for `path_filter` yielding matching paths, and not values or keys, is that if you have the paths, you can than get the keys and values with them, using `path_get`. ```pycon >>> from functools import partial, reduce >>> path_get = lambda m, k: reduce(lambda m, k: m[k], k, m) >>> extract_paths = lambda m, paths: map(partial(path_get, m), paths) >>> vals = list(extract_paths(mm, paths)) >>> vals [42, 'meaning of life'] ``` #### NOTE pkv_filt is first to match the order of the arguments of the builtin filter function. ### dol.paths.path_get(obj, path, on_error=, \*, sep=None, key_transformer=None, get_value=, caught_errors=(, )) Get elements of a mapping through a path to be called recursively. * **Parameters:** * **obj** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The object to get the path from * **path** – The path to get * **on_error** (`Union`[[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The error handler to use (default: raise_on_error) * **sep** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Determines a path is transforms into a tuple of keys. If it’s a string, `lambda path: path.split(sep)` is used. If not, it should be a function which takes in a path object and returns an iterable of keys. * **key_transformer** – A function to transform the keys of the path * **get_value** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – A function to get the value of a key in a mapping * **caught_errors** – The errors to catch (default: Exception) It will - split a path into keys (if sep is given, or if path is a string, will use ‘.’ as a separator by default) - if key_transformer is given, apply to each key - consider string keys that are numeric as ints (convenient for lists) - get items also as attributes (attributes are checked for first for string keys) - catch all exceptions (that are subclasses of `Exception`) ```pycon >>> class A: ... an_attribute = 42 >>> path_get([1, [4, 5, {'a': A}], 3], [1, 2, 'a', 'an_attribute']) 42 ``` By default, if `path` is a string, it will be split on `sep`, which is `'.'` by default. ```pycon >>> path_get([1, [4, 5, {'a': A}], 3], '1.2.a.an_attribute') 42 ``` #### NOTE The underlying function is `_path_get`, but `path_get` has defaults and flexible input processing for more convenience. #### NOTE `path_get` contains some ready-made `OnErrorType` functions in its attributes. For example, see how we can make `path_get` have the same behavior as `dict.get` by passing `path_get.return_none_on_error` as `on_error`: ```pycon >>> dd = path_get({}, 'no.keys', on_error=path_get.return_none_on_error) >>> dd is None True ``` For example, `path_get.raise_on_error`, `path_get.return_none_on_error`, and `path_get.return_empty_tuple_on_error`. ### dol.paths.paths_getter(paths, obj=None, \*, egress=, on_error=, sep=None, key_transformer=None, get_value=, caught_errors=(, )) Returns (path, values) pairs of the given paths in the given object. This is the “fan-out” version of `path_get`, specifically designed to get multiple paths, returning the (path, value) pairs in a dict (by default), or via any pairs aggregator (`egress`) function. #### NOTE For reasons who’s clarity is burried in historical legacy, the order of obj and path are the opposite of path_get. * **Parameters:** * **paths** – The paths to get * **obj** – The object to get the paths from * **egress** – The egress function to use (default: dict) * **on_error** (`Union`[[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The error handler to use (default: raise_on_error) * **sep** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The separator to use if the path is a string * **key_transformer** – A function to transform the keys of the path * **get_value** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – A function to get the value of a key in a mapping * **caught_errors** – The errors to catch (default: Exception) ```pycon >>> obj = {'a': {'b': 1, 'c': 2}, 'd': 3} >>> paths = ['a.c', 'd'] >>> paths_getter(paths, obj=obj) {'a.c': 2, 'd': 3} >>> path_extractor = paths_getter(paths) >>> path_extractor(obj) {'a.c': 2, 'd': 3} ``` See that the paths are used as the keys of the returned dict. If you want to specify your own keys, you can simply specify `paths` as a dict whose keys are the keys you want, and whose values are the paths to get: ```pycon >>> path_extractor_2 = paths_getter({'california': 'a.c', 'dreaming': 'd'}) >>> path_extractor_2(obj) {'california': 2, 'dreaming': 3} ``` ### dol.paths.prefixless_view(store=None, , prefix=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Wrap `store` so that keys are seen without `prefix` (added back on access). ### dol.paths.raise_on_error(d) `on_error` policy for `path_get`: re-raise the caught error. ### dol.paths.rel_path_wrap(o, \_prefix) * **Parameters:** * **o** – An object to be wrapped * **\_prefix** – The \_prefix to use for key wrapping (will remove it from outcoming keys and add to ingoing keys. ```pycon >>> # The dynamic way (if you try this at home, be aware of the pitfalls of the dynamic way >>> # -- but don't just believe the static dogmas). >>> d = {'/ROOT/of/every/thing': 42, '/ROOT/of/this/too': 0} >>> dd = rel_path_wrap(d, '/ROOT/of/') >>> dd['foo'] = 'bar' >>> dict(dd.items()) # gives us what you would expect {'every/thing': 42, 'this/too': 0, 'foo': 'bar'} >>> # but under the hood, the dict we wrapped actually contains the '/ROOT/' prefix >>> dict(dd.store) {'/ROOT/of/every/thing': 42, '/ROOT/of/this/too': 0, '/ROOT/of/foo': 'bar'} >>> >>> # The static way: Make a class that will integrate the _prefix at construction time. >>> class MyStore(mk_relative_path_store(dict)): # Indeed, mk_relative_path_store(dict) is a class you can subclass ... def __init__(self, _prefix, *args, **kwargs): ... self._prefix = _prefix ``` ### dol.paths.return_empty_tuple_on_error(d) `on_error` policy for `path_get`: return `()`. ### dol.paths.return_none_on_error(d) `on_error` policy for `path_get`: return `None`. ### dol.paths.search_paths(d, pkv_filt, , leafs_only=True, breadth_first=False) backwards compatibility quasi-alias (arguments are flipped) Use path_filter instead, since search_paths will be deprecated. * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`)] ### dol.paths.separate_keys_with_separator(obj, sep='.') Split a string path on `sep` and cast numeric parts to `int`; a non-string iterable is only cast element-wise. ### dol.paths.separator_based_path_extender(path, key, sep) Extends a given path with a new key using the specified separator. If the path is empty, the key is returned as is. * **Return type:** [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Path`) ### dol.paths.split_if_str(obj, sep='.') Split `obj` on `sep` if it is a string; return it unchanged otherwise. ### dol.paths.str_template_key_trans(template, key_type, format_dict=None, process_kwargs=None, process_info_dict=None, named_tuple_type_name='NamedTuple', sep='/') Make a key trans object that translates from a string \_id to a dict, tuple, or namedtuple key (and back) ### dol.paths.string_unparse(parsing_result) The inverse of string.Formatter.parse Will ravel ```pycon >>> import string >>> formatter = string.Formatter() >>> string_unparse(formatter.parse('literal{name!c:spec}')) 'literal{name!c:spec}' ``` # _autosummary/dol.recipes.html.md # dol.recipes Recipes using dol ### Functions | [`search_paths`](_autosummary/dol.recipes.html.md#dol.recipes.search_paths)(pkv_filt, d, \*[, leafs_only, ...]) | Walk a dict, yielding paths to values that pass the `pkv_filt` | |-----------------------------------------------------------------------------------------------------|------------------------------------------------------------------| ### dol.recipes.search_paths(pkv_filt, d, , leafs_only=True, breadth_first=False) Walk a dict, yielding paths to values that pass the `pkv_filt` * **Parameters:** * **pkv_filt** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – A function that takes a path, key, and value, and returns `True` if the path should be yielded, and `False` otherwise * **d** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – The `Mapping` to walk (scan through) * **leafs_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to yield only paths to leafs (default), or to yield paths to all values that pass the `pkv_filt`. * **breadth_first** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to perform breadth-first traversal (instead of the default depth-first traversal). * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`PT`)] * **Returns:** An iterator of paths to values that pass the `pkv_filt` ### Example ```pycon >>> d = {'a': {'b': {'c': 1, 'd': 2}, 'e': 3}} >>> list(path_filter(lambda p, k, v: v == 2, d)) [('a', 'b', 'd')] ``` ```pycon >>> mm = { ... 'a': {'b': {'c': 42}}, ... 'aa': {'bb': {'cc': 'meaning of life'}}, ... 'aaa': {'bbb': 314}, ... } >>> return_path_if_int_leaf = lambda p, k, v: (p, v) if isinstance(v, int) else None >>> paths = list(path_filter(return_path_if_int_leaf, mm)) >>> paths # only the paths to the int leaves are returned [('a', 'b', 'c'), ('aaa', 'bbb')] ``` The `pkv_filt` argument can use path, key, and/or value to define your search query. For example, let’s extract all the paths that have depth at least 3. ```pycon >>> paths = list(path_filter(lambda p, k, v: len(p) >= 3, mm)) >>> paths [('a', 'b', 'c'), ('aa', 'bb', 'cc')] ``` The rationale for `path_filter` yielding matching paths, and not values or keys, is that if you have the paths, you can than get the keys and values with them, using `path_get`. ```pycon >>> from functools import partial, reduce >>> path_get = lambda m, k: reduce(lambda m, k: m[k], k, m) >>> extract_paths = lambda m, paths: map(partial(path_get, m), paths) >>> vals = list(extract_paths(mm, paths)) >>> vals [42, 'meaning of life'] ``` #### NOTE pkv_filt is first to match the order of the arguments of the builtin filter function. # _autosummary/dol.signatures.html.md # dol.signatures Signature calculus: Tools to make it easier to work with function’s signatures. How to: > - get names, kinds, defaults, annotations > - make signatures flexibly > - merge two or more signatures > - give a function a specific signature (with a choice of validations) > - get an equivalent function with a different order of arguments > - get an equivalent function with a subset of arguments (like partial) > - get an equivalent function but with variadic `*args` and/or `**kwargs` replaced with > non-variadic args (tuple) and kwargs (dict) > - make an f(a) function in to a f(a, b=None) function with b ignored Get names, kinds, defaults, annotations: ```pycon >>> def func(z, a: float=1.0, /, b=2, *, c: int=3): ... pass >>> sig = Sig(func) >>> sig.names ['z', 'a', 'b', 'c'] >>> from inspect import Parameter >>> assert sig.kinds == { ... 'z': Parameter.POSITIONAL_ONLY, ... 'a': Parameter.POSITIONAL_ONLY, ... 'b': Parameter.POSITIONAL_OR_KEYWORD, ... 'c': Parameter.KEYWORD_ONLY ... } >>> # Note z is not in there (only defaulted params are included) >>> sig.defaults {'a': 1.0, 'b': 2, 'c': 3} >>> sig.annotations {'a': , 'c': } ``` Make signatures flexibly: ```pycon >>> Sig(func) >>> Sig(['a', 'b']) >>> Sig('x y z') ``` Merge signatures. ```pycon >>> def foo(x): pass >>> def bar(y: int, *, z=2): pass # note the * (keyword only) will be lost! >>> Sig(foo) + ['a', 'b'] + Sig(bar) ``` Give a function a signature. ```pycon >>> @Sig('a b c') ... def func(*args, **kwargs): ... print(args, kwargs) >>> Sig(func) ``` **Notes to the reader** Both in the code and in the docs, we’ll use short hands for parameter (argument) kind. > - PK = Parameter.POSITIONAL_OR_KEYWORD > - VP = Parameter.VAR_POSITIONAL > - VK = Parameter.VAR_KEYWORD > - PO = Parameter.POSITIONAL_ONLY > - KO = Parameter.KEYWORD_ONLY ### Functions | [`all_pk_signature`](_autosummary/dol.signatures.html.md#dol.signatures.all_pk_signature)(callable_or_signature) | Changes all (non-variadic) arguments to be of the PK (POSITION_OR_KEYWORD) kind. | |-----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| | `assure_callable`(obj) | | | [`assure_params`](_autosummary/dol.signatures.html.md#dol.signatures.assure_params)([obj]) | Get an interable of Parameter instances from an object. | | `assure_signature`(obj) | | | [`call_forgivingly`](_autosummary/dol.signatures.html.md#dol.signatures.call_forgivingly)(func, \*args, \*\*kwargs) | Call function on given args and kwargs, but only taking what the function needs (not choking if they're extras variables) | | [`call_somewhat_forgivingly`](_autosummary/dol.signatures.html.md#dol.signatures.call_somewhat_forgivingly)(func, args, kwargs) | Call function on given args and kwargs, but with controllable argument leniency. | | [`ch_func_to_all_pk`](_autosummary/dol.signatures.html.md#dol.signatures.ch_func_to_all_pk)(func) | Returns a decorated function where all arguments are of the PK kind. | | [`ch_signature_to_all_pk`](_autosummary/dol.signatures.html.md#dol.signatures.ch_signature_to_all_pk)(callable_or_signature) | Changes all (non-variadic) arguments to be of the PK (POSITION_OR_KEYWORD) kind. | | [`ch_variadics_to_non_variadic_kind`](_autosummary/dol.signatures.html.md#dol.signatures.ch_variadics_to_non_variadic_kind)(func, \*[, ...]) | A decorator that will change a VAR_POSITIONAL (`*args`) argument to a tuple (args) argument of the same name. | | [`common_and_diff_argnames`](_autosummary/dol.signatures.html.md#dol.signatures.common_and_diff_argnames)(func1, func2) | Get list of argument names that are common to two functions, as well as the two lists of names that are different | | `compare_signatures`(func1, func2[, ...]) | | | `convert_to_PK`(kinds) | | | [`copy_func`](_autosummary/dol.signatures.html.md#dol.signatures.copy_func)(f) | Copy a function (not sure it works with all types of callables) | | [`defaults_are_the_same_when_not_empty`](_autosummary/dol.signatures.html.md#dol.signatures.defaults_are_the_same_when_not_empty)(dflt1, ...) | Check if two defaults are the same when they are not empty. | | `deprecation_of`(func, old_name) | | | [`dflt1_is_empty_or_dflt2_is_not`](_autosummary/dol.signatures.html.md#dol.signatures.dflt1_is_empty_or_dflt2_is_not)(dflt1, dflt2) | Why such a strange default comparison function? | | [`dict_of_attribute_signatures`](_autosummary/dol.signatures.html.md#dol.signatures.dict_of_attribute_signatures)(cls) | A function that extracts the signatures of all callable attributes of a class. | | `ensure_callable`(obj) | | | `ensure_param`(p) | | | [`ensure_params`](_autosummary/dol.signatures.html.md#dol.signatures.ensure_params)([obj]) | Get an interable of Parameter instances from an object. | | `ensure_signature`(obj) | | | `expand_nested_key`(d, k) | | | [`extract_arguments`](_autosummary/dol.signatures.html.md#dol.signatures.extract_arguments)(params, \*[, ...]) | Extract arguments needed to satisfy the params of a callable, dealing with the dirty details. | | `flatten_if_var_kw`(kvs, var_kw_name) | | | `function_caller`(func, args, kwargs) | | | [`has_signature`](_autosummary/dol.signatures.html.md#dol.signatures.has_signature)(obj[, robust]) | Check if an object has a signature -- i.e. is callable and inspect.signature( obj) returns something. | | `ignore_any_differences`(x, y) | | | [`insert_annotations`](_autosummary/dol.signatures.html.md#dol.signatures.insert_annotations)(s, /, \*, ...) | Insert annotations in a signature. | | [`is_call_compatible_with`](_autosummary/dol.signatures.html.md#dol.signatures.is_call_compatible_with)(sig1, sig2, \*[, ...]) | Return True if `sig1` is compatible with `sig2`. | | [`is_signature_error`](_autosummary/dol.signatures.html.md#dol.signatures.is_signature_error)(e) | Check if an exception is a signature error | | [`keyed_comparator`](_autosummary/dol.signatures.html.md#dol.signatures.keyed_comparator)(comparator, key) | Create a key-function enabled binary operator. | | [`kind_forgiving_func`](_autosummary/dol.signatures.html.md#dol.signatures.kind_forgiving_func)(func[, kinds_modifier]) | Wraps the func, changing the argument kinds according to kinds_modifier. | | `maybe_first`(items) | | | `mk_func_comparator_based_on_signature_comparator`(...) | | | [`mk_sig_from_args`](_autosummary/dol.signatures.html.md#dol.signatures.mk_sig_from_args)(\*args_without_default, ...) | Make a Signature instance by specifying args_without_default and args_with_defaults. | | [`name_of_obj`](_autosummary/dol.signatures.html.md#dol.signatures.name_of_obj)(o, \*[, base_name_of_obj, ...]) | Tries to find the (or "a") name for an object, even if `__name__` doesn't exist. | | `name_of_var_kw_argument`(sig) | | | `normalized_func`(func) | | | `param_attribute_dict`(...) | | | [`param_binary_func`](_autosummary/dol.signatures.html.md#dol.signatures.param_binary_func)(param1, param2, \*[, name, ...]) | Compare two parameters. | | [`param_comparator`](_autosummary/dol.signatures.html.md#dol.signatures.param_comparator)(param1, param2, \*[, name, ...]) | Compare two parameters. | | [`param_differences_dict`](_autosummary/dol.signatures.html.md#dol.signatures.param_differences_dict)(param1, param2, \*[, ...]) | Makes a dictionary exibiting the differences between two parameters. | | [`param_for_kind`](_autosummary/dol.signatures.html.md#dol.signatures.param_for_kind)([name, kind, with_default]) | Function to easily and flexibly make inspect.Parameter objects for testing. | | `param_has_default_or_is_var_kind`(p) | | | `parameter_to_dict`(p) | | | `params_of`(obj) | | | [`postprocess`](_autosummary/dol.signatures.html.md#dol.signatures.postprocess)(egress) | A decorator that will process the output of the wrapped function with egress | | [`replace_kwargs_using`](_autosummary/dol.signatures.html.md#dol.signatures.replace_kwargs_using)(sig) | Decorator that replaces the variadic keyword argument of the target function using the `sig`, the signature of a source function. | | [`resolve_function`](_autosummary/dol.signatures.html.md#dol.signatures.resolve_function)(obj) | Get the underlying function of a property or cached_property | | `return_tuple`(x, y) | | | [`set_signature_of_func`](_autosummary/dol.signatures.html.md#dol.signatures.set_signature_of_func)(func, parameters, \*, ...) | Set the signature of a function, with sugar. | | [`sig_to_dataclass`](_autosummary/dol.signatures.html.md#dol.signatures.sig_to_dataclass)(sig, \*[, cls_name, bases, ...]) | Make a `class` (through `make_dataclass`) from the given signature. | | [`sort_params`](_autosummary/dol.signatures.html.md#dol.signatures.sort_params)(params) | | | [`use_interface`](_autosummary/dol.signatures.html.md#dol.signatures.use_interface)(interface_sig) | Use interface_sig as (enforced/validated) signature of the decorated function. | | [`validate_signature`](_autosummary/dol.signatures.html.md#dol.signatures.validate_signature)(func) | Validates the signature of a function. | ### Classes | [`MissingArgValFor`](_autosummary/dol.signatures.html.md#dol.signatures.MissingArgValFor)(argname) | A simple class to wrap an argument name, indicating that it was missing somewhere. | |-----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`P`](_autosummary/dol.signatures.html.md#dol.signatures.P) | | | [`Param`](_autosummary/dol.signatures.html.md#dol.signatures.Param)(name[, kind]) | A thin wrap of Parameters: Adds shorter aliases to argument kinds and a POSITIONAL_OR_KEYWORD default to the argument kind to make it faster to make Parameter objects | | [`Sig`](_autosummary/dol.signatures.html.md#dol.signatures.Sig)([obj, name, \_\_validate_parameters_\_]) | A subclass of inspect.Signature that has a lot of extra api sugar, such as | | [`SigPair`](_autosummary/dol.signatures.html.md#dol.signatures.SigPair)(sig1, sig2) | Class that operates on a pair of signatures. | ### Exceptions | [`FuncCallNotMatchingSignature`](_autosummary/dol.signatures.html.md#dol.signatures.FuncCallNotMatchingSignature) | Raise when the call signature is not valid | |-----------------------------------------------------------------------------------------------|----------------------------------------------| | [`IncompatibleSignatures`](_autosummary/dol.signatures.html.md#dol.signatures.IncompatibleSignatures)(\*args[, sig1, sig2]) | | | [`InvalidSignature`](_autosummary/dol.signatures.html.md#dol.signatures.InvalidSignature) | Raise when a signature is not valid | ### *exception* dol.signatures.FuncCallNotMatchingSignature Bases: [`TypeError`](https://docs.python.org/3/builtins/exceptions.html#TypeError) Raise when the call signature is not valid ### *exception* dol.signatures.IncompatibleSignatures(\*args, sig1=None, sig2=None, \*\*kwargs) Bases: [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) #### pformat(indent=1, width=80, depth=None, , compact=False, sort_dicts=True, underscore_numbers=False) Format a Python object into a pretty-printed representation. ### *exception* dol.signatures.InvalidSignature Bases: [`SyntaxError`](https://docs.python.org/3/builtins/exceptions.html#SyntaxError), [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) Raise when a signature is not valid ### *class* dol.signatures.MissingArgValFor(argname) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A simple class to wrap an argument name, indicating that it was missing somewhere. ```pycon >>> MissingArgValFor("argname") MissingArgValFor("argname") ``` ### dol.signatures.P alias of [`Param`](_autosummary/dol.signatures.html.md#dol.signatures.Param) ### *class* dol.signatures.Param(name, kind=\_ParameterKind.POSITIONAL_OR_KEYWORD, , default, annotation) Bases: [`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter) A thin wrap of Parameters: Adds shorter aliases to argument kinds and a POSITIONAL_OR_KEYWORD default to the argument kind to make it faster to make Parameter objects ```pycon >>> list(map(Param, 'some quick arg params'.split())) [, , , ] >>> from inspect import Signature >>> P = Param >>> Signature([P('x', P.PO), P('y', default=42, annotation=int), P('kw', P.KO)]) ``` ### *class* dol.signatures.Sig(obj=None, , name=None, return_annotation, \_\_validate_parameters_\_=True) Bases: [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) A subclass of inspect.Signature that has a lot of extra api sugar, such as - making a signature for a variety of input types (callable, iterable of callables, parameter lists, strings, etc.) - has a dict-like interface - signature merging (with operator interfaces) - quick access to signature data - positional/keyword argument mapping. ### Positional/Keyword argument mapping In python, arguments can be positional (args) or keyword (kwargs). … sometimes both, sometimes a single one is imposed. … and you have variadic versions of both. … and you can have defaults or not. … and all these different kinds have a particular order they must be in. It’s is mess really. The flexibility is nice – but still; a mess. You only really feel the mess if you try to do some meta-programming with your functions. Then, methods like `normalize_kind` can help you out, since you can enforce, and then assume, some stable interface to your functions. Two of the base methods for dealing with positional (args) and keyword (kwargs) inputs are: > - `map_arguments`: Map some args/kwargs input to a keyword-only > expression of the inputs. This is useful if you need to do some processing > based on the argument names. > - `mk_args_and_kwargs`: Translate a fully keyword expression of some > inputs into an (args, kwargs) pair that can be used to call the function. > (Remember, your function can have constraints, so you may need to do this. The usual pattern of use of these methods is to use `map_arguments` to map all the inputs to their corresponding name, do what needs to be done with that (example, validation, transformation, decoration…) and then map back to an (args, kwargs) pair than can actually be used to call the function. Examples of methods and functions using these: `call_forgivingly`, `tuple_the_args`, `map_arguments_from_variadics`, `extract_args_and_kwargs`, `source_arguments`, and `source_args_and_kwargs`. ### Making a signature You can construct a `Sig` object from a callable, ```pycon >>> def f(w, /, x: float = 1, y=1, *, z: int = 1): ... ... >>> Sig(f) ``` but also from any “ParamsAble” object. Such as… an iterable of Parameter instances, strings, tuples, or dicts: ```pycon >>> Sig( ... [ ... "a", ... ("b", Parameter.empty, int), ... ("c", 2), ... ("d", 1.0, float), ... dict(name="special", kind=Parameter.KEYWORD_ONLY, default=0), ... ] ... ) >>> >>> Sig( ... [ ... "a", ... "b", ... dict(name="args", kind=Parameter.VAR_POSITIONAL), ... dict(name="kwargs", kind=Parameter.VAR_KEYWORD), ... ] ... ) ``` The parameters of a signature are like a matrix whose rows are the parameters, and the 4 columns are their properties: name, kind, default, and annotation (the two laste ones being optional). You get a row view when doing `Sig(...).parameters.values()`, but what if you want a column-view? Here’s how: ```pycon >>> def f(w, /, x: float = 1, y=2, *, z: int = 3): ... ... >>> >>> s = Sig(f) >>> s.kinds {'w': <_ParameterKind.POSITIONAL_ONLY: 0>, 'x': <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, 'y': <_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, 'z': <_ParameterKind.KEYWORD_ONLY: 3>} ``` ```pycon >>> s.annotations {'x': , 'z': } >>> assert ( ... s.annotations == f.__annotations__ ... ) # same as what you get in `__annotations__` >>> >>> s.defaults {'x': 1, 'y': 2, 'z': 3} >>> # Note that it's not the same as you get in __defaults__ though: >>> assert ( ... s.defaults != f.__defaults__ == (1, 2) ... ) # not 3, since __kwdefaults__ has that! ``` We can sum (i.e. merge) and subtract (i.e. remove arguments) Sig instances. Also, Sig instance is callable. It has the effect of inserting it’s signature in the input (in `__signature__`, but also inserting the resulting `__defaults__` and `__kwdefaults__`). One of the intents is to be able to do things like: ```pycon >>> import inspect >>> def f(w, /, x: float = 1, y=1, *, z: int = 1): ... ... >>> def g(i, w, /, j=2): ... ... ... >>> >>> @Sig.from_objs(f, g, ["a", ("b", 3.14), ("c", 42, int)]) ... def some_func(*args, **kwargs): ... ... >>> inspect.signature(some_func) >>> >>> sig = Sig(f) + g + ["a", ("b", 3.14), ("c", 42, int)] - "b" - ["a", "z"] >>> @sig ... def some_func(*args, **kwargs): ... ... >>> inspect.signature(some_func) ``` #### add_optional_keywords(kwarg_and_defaults=None, kwarg_annotations=None) Add optional keyword arguments to a signature. ```pycon >>> @Sig.add_optional_keywords({"c": 2, "d": 3}, {"c": int}) ... def foo(a, *, b=1, **kwargs): ... return f"{a=}, {b=}, {kwargs=}" ... ``` You can still call the function as before, and like before, any “extra” keyword arguments will be passed to kwargs: ```pycon >>> foo(0, d=10) "a=0, b=1, kwargs={'d': 10}" ``` The difference is that now the signature of `foo` now has `c` and `d`: ```pycon >>> str(Sig(foo)) '(a, *, c: int = 2, d=3, b=1, **kwargs)' ``` #### add_params(params) Creates a new instance of Sig after merging the parameters of this signature with a list of new parameters. The new list of parameters is automatically sorted based on signature constraints given by kinds and default values. See Python native signature documentation for more details. ```pycon >>> s = Sig('(a, /, b, *, c)') >>> s.add_params([ ... Param('kwargs', VK), ... dict(name='d', kind=KO), ... Param('args', VP), ... 'e', ... Param('f', PO), ... ]) ``` #### *property* annotations annotation, …} dict of annotations of the signature. What `func.__annotations__` would give you. * **Type:** {arg_name #### args_and_kwargs_from_kwargs(arguments, , apply_defaults=False, allow_partial=False, allow_excess=False, ignore_kind=False, args_limit=0) Extract args and kwargs such that `func(*args, **kwargs)` can be called, where func has instance’s signature. * **Parameters:** * **arguments** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – The {param_name: arg_val,…} dict to process * **args_limit** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – How “far” in the params should args (positional arguments) be searched for. - args_limit==0: Take the minimum number possible of args (positional arguments). Only those that are position only or before a var-positional. - args_limit is None: Take the maximum number of args (positional arguments). The only kwargs (keyword arguments) you should have are keyword-only and var-keyword arguments. - args_limit positive integer: Take the args_limit first argument names (of signature) as args, and the rest as kwargs. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ```pycon >>> def foo(w, /, x: float, y=1, *, z: int = 1): ... return ((w + x) * y) ** z >>> foo_sig = Sig(foo) >>> args, kwargs = foo_sig.mk_args_and_kwargs( ... dict(w=4, x=3, y=2, z=1) ... ) >>> assert (args, kwargs) == ((4,), {"x": 3, "y": 2, "z": 1}) >>> assert foo(*args, **kwargs) == foo(4, 3, 2, z=1) == 14 ``` What about variadics? ```pycon >>> def bar(a, /, b, *args, c=2, **kwargs): ... pass >>> Sig(bar).mk_args_and_kwargs( ... dict(a=1, b=2, args=(3,4), c=5, kwargs=dict(d=6, e=7)) ... ) ((1, 2, 3, 4), {'c': 5, 'd': 6, 'e': 7}) ``` You can also give the arguments in a different order: ```pycon >>> Sig(bar).mk_args_and_kwargs( ... dict(args=(3,4), kwargs=dict(d=6, e=7), b=2, c=5, a=1) ... ) ((1, 2, 3, 4), {'c': 5, 'd': 6, 'e': 7}) ``` The `args_limit` begs explanation. Consider the signature of `def foo(w, /, x: float, y=1, *, z: int = 1): ...` for instance. We could call the function with the following (args, kwargs) pairs: - ((1,), {‘x’: 2, ‘y’: 3, ‘z’: 4}) - ((1, 2), {‘y’: 3, ‘z’: 4}) - ((1, 2, 3), {‘z’: 4}) The two other combinations (empty args or empty kwargs) are not valid because of the / and \* constraints. But when asked for an (args, kwargs) pair, which of the three valid options should be returned? This is what the `args_limit` argument controls. If `args_limit == 0`, the least args (positional arguments) will be returned. It’s the default. ```pycon >>> arguments = dict(w=4, x=3, y=2, z=1) >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=0) ((4,), {'x': 3, 'y': 2, 'z': 1}) ``` If `args_limit is None`, the least kwargs (keyword arguments) will be returned. ```pycon >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=None) ((4, 3, 2), {'z': 1}) ``` If `args_limit` is a positive integer, the first `[args_limit]` arguments will be returned (not checking at all if this is valid!). ```pycon >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=1) ((4,), {'x': 3, 'y': 2, 'z': 1}) >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=2) ((4, 3), {'y': 2, 'z': 1}) >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=3) ((4, 3, 2), {'z': 1}) ``` Note that if you specify `args_limit` to be greater than the maximum of positional arguments, it behaves as if `args_limit` was `None`: ```pycon >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=4) ((4, 3, 2), {'z': 1}) ``` Note that ‘args_limit’’s behavior is consistent with list behvior in the sense that: ```pycon >>> args = (0, 1, 2, 3) >>> args[:0] () >>> args[:None] (0, 1, 2, 3) >>> args[2] 2 ``` If variable positional arguments are present, `args_limit` is ignored and all positional arguments are returned as args. ```pycon >>> Sig(bar).mk_args_and_kwargs( ... dict(a=1, b=2, args=(3,4), c=5, kwargs=dict(d=6, e=7)), ... args_limit=1 ... ) ((1, 2, 3, 4), {'c': 5, 'd': 6, 'e': 7}) ``` By default, only the arguments that were given in the `arguments` input will be returned in the (args, kwargs) output. If you also want to get those that have defaults (according to signature), you need to specify it with the `apply_defaults=True` argument. ```pycon >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3)) ((4,), {'x': 3}) >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3), apply_defaults=True) ((4,), {'x': 3, 'y': 1, 'z': 1}) ``` By default, all required arguments must be given. Not doing so will lead to a `TypeError`. If you want to process your arguments anyway, specify `allow_partial=True`. ```pycon >>> foo_sig.mk_args_and_kwargs(dict(w=4)) Traceback (most recent call last): ... TypeError: missing a required argument: 'x' >>> foo_sig.mk_args_and_kwargs(dict(w=4), allow_partial=True) ((4,), {}) ``` Specifying argument names that are not recognized by the signature will lead to a `TypeError`. If you want to avoid this (and just take from the input `kwargs` what ever you can), specify this with `allow_excess=True`. ```pycon >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3, extra='stuff')) Traceback (most recent call last): ... TypeError: Got unexpected keyword arguments: extra >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3, extra='stuff'), ... allow_excess=True) ((4,), {'x': 3}) ``` See `map_arguments` (namely for the description of the arguments). #### ch_param_attrs(param_attr, \*arg_new_vals, \_allow_reordering=False, \*\*kwargs_new_vals) Change a specific attribute of the params, returning a modified signature. This is a convenience method for the modified method when we’re targetting a fixed param attribute: ‘name’, ‘kind’, ‘default’, or ‘annotation’ Instead of having to do this ```pycon >>> def foo(a, *b, **c): ... >>> Sig(foo).modified(a={'name': 'A'}, b={'name': 'B'}, c={'name': 'C'}) ``` We can simply do this ```pycon >>> Sig(foo).ch_param_attrs('name', a='A', b='B', c='C') ``` One quite useful thing you can do with this is to set defaults, or set defaults where there are none. If you wrap your function with such a modified signature, you get a “curried” version of your function (called “partial” in python). (Note that the `functools.wraps` won’t deal with defaults “correctly”, but wrapping with `Sig` objects takes care of that oversight!) ```pycon >>> def foo(a, b, c): ... return a + b * c >>> special_foo = Sig(foo).ch_param_attrs('default', b=2, c=3)(foo) >>> Sig(special_foo) >>> special_foo(5) # should be 5 + 2 * 3 == 11 11 ``` #### *property* defaults A `{name: default,...}` dict of defaults (regardless of kind) #### extract_args_and_kwargs(\*args, \_ignore_kind=True, \_allow_partial=False, \_allow_excess=True, \_apply_defaults=False, \_args_limit=0, \*\*kwargs) Source the (args, kwargs) for the signature instance, ignoring excess arguments. ```pycon >>> def foo(w, /, x: float, y=2, *, z: int = 1): ... return w + x * y ** z >>> args, kwargs = Sig(foo).extract_args_and_kwargs(4, x=3, y=2) >>> (args, kwargs) == ((4,), {"x": 3, "y": 2}) True ``` The difference with map_arguments_from_variadics is that here the output is ready to be called by the function whose signature we have, since the position-only arguments will be returned as args. ```pycon >>> foo(*args, **kwargs) 10 ``` Note that though `w` is a position only argument, you can specify `w=4` as a keyword argument too (by default): ```pycon >>> args, kwargs = Sig(foo).extract_args_and_kwargs(w=4, x=3, y=2) >>> (args, kwargs) == ((4,), {"x": 3, "y": 2}) True ``` If you don’t want to allow that, you can say `_ignore_kind=False` ```pycon >>> Sig(foo).extract_args_and_kwargs(w=4, x=3, y=2, _ignore_kind=False) Traceback (most recent call last): ... TypeError:...'w'... ``` You can use `_allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> Sig(foo).extract_args_and_kwargs(x=3, y=2) Traceback (most recent call last): ... TypeError:...'w'... ``` But if you specify `_allow_partial=True`… ```pycon >>> args, kwargs = Sig(foo).extract_args_and_kwargs( ... x=3, y=2, _allow_partial=True ... ) >>> (args, kwargs) == ((), {"x": 3, "y": 2}) True ``` By default, `_apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> args, kwargs = Sig(foo).extract_args_and_kwargs(4, x=3, y=2) >>> (args, kwargs) == ((4,), {"x": 3, "y": 2}) True ``` But if you specify `_apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> args, kwargs = Sig(foo).extract_args_and_kwargs( ... 4, x=3, y=2, _apply_defaults=True ... ) >>> (args, kwargs) == ((4,), {"x": 3, "y": 2, "z": 1}) True ``` #### extract_kwargs(\*args, \_apply_defaults=False, \_allow_partial=False, \_allow_excess=False, \_ignore_kind=False, \*\*kwargs) Convenience method that calls map_arguments from variadics ```pycon >>> def foo(w, /, x: float, y="YY", *, z: str = "ZZ"): ... ... >>> sig = Sig(foo) >>> assert ( ... sig.map_arguments_from_variadics(1, 2, 3, z=4) ... == sig.map_arguments_from_variadics(1, 2, y=3, z=4) ... == {"w": 1, "x": 2, "y": 3, "z": 4} ... ) ``` What about var positional and var keywords? ```pycon >>> def bar(*args, **kwargs): ... ... ... >>> Sig(bar).map_arguments_from_variadics(1, 2, y=3, z=4) {'args': (1, 2), 'kwargs': {'y': 3, 'z': 4}} ``` Note that though `w` is a position only argument, you can specify `w=11` as a keyword argument too, using `_ignore_kind=True`: ```pycon >>> Sig(foo).map_arguments_from_variadics(w=11, x=22, _ignore_kind=True) {'w': 11, 'x': 22} ``` You can use `_allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> Sig(foo).map_arguments_from_variadics(x=3, y=2) Traceback (most recent call last): ... TypeError: missing a required argument: 'w' ``` But if you specify `_allow_partial=True`… ```pycon >>> Sig(foo).map_arguments_from_variadics(x=3, y=2, _allow_partial=True) {'x': 3, 'y': 2} ``` By default, `_apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> Sig(foo).map_arguments_from_variadics(4, x=3, y=2) {'w': 4, 'x': 3, 'y': 2} ``` But if you specify `_apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> Sig(foo).map_arguments_from_variadics(4, x=3, y=2, _apply_defaults=True) {'w': 4, 'x': 3, 'y': 2, 'z': 'ZZ'} ``` #### get_names(spec, , conserve_sig_order=True, allow_excess=False) Return a tuple of names corresponding to the given spec. * **Parameters:** * **spec** – An integer, string, or iterable of intergers and strings * **conserve_sig_order** – Whether to order according to the signature * **allow_excess** – Whether to allow items in spec that are not in signature ```pycon >>> sig = Sig('a b c d e') >>> sig.get_names(0) ('a',) >>> sig.get_names([0, 2]) ('a', 'c') >>> sig.get_names('b') ('b',) >>> sig.get_names([0, 'c', -1]) ('a', 'c', 'e') ``` See that by default the order of the signature is conserved: ```pycon >>> sig.get_names('b e d') ('b', 'd', 'e') ``` But you can change that default to conserve the order of the `spec` instead: ```pycon >>> sig.get_names('b e d', conserve_sig_order=False) ('b', 'e', 'd') ``` By default, you can’t mention names that are not in signature. To allow this (making `spec` have “extract these” interpretation), set `allow_excess=True`: ```pycon >>> sig.get_names(['a', 'c', 'e', 'g', 'h'], allow_excess=True) ('a', 'c', 'e') ``` #### *property* has_var_keyword Use index_of_var_keyword or var_keyword_name directly when needing that information as well. This will avoid having to check the kinds list twice. #### *property* has_var_kinds Whether the signature has a VAR_POSITIONAL or a VAR_KEYWORD parameter. ```pycon >>> Sig(lambda x, *, y: None).has_var_kinds False >>> Sig(lambda x, *y: None).has_var_kinds True >>> Sig(lambda x, **y: None).has_var_kinds True ``` #### *property* has_var_positional Use index_of_var_positional or var_keyword_name directly when needing that information as well. This will avoid having to check the kinds list twice. #### *property* index_of_var_keyword The index of a VAR_KEYWORD param kind if any, and None if not. See also, Sig.index_of_var_positional ```pycon >>> assert Sig(lambda **kwargs: 0).index_of_var_keyword == 0 >>> assert Sig(lambda a, **kwargs: 0).index_of_var_keyword == 1 >>> assert Sig(lambda a, *args, **kwargs: 0).index_of_var_keyword == 2 ``` And if there’s none… ```pycon >>> assert Sig(lambda a, *args, b=1: 0).index_of_var_keyword is None ``` #### *property* index_of_var_positional The index of the VAR_POSITIONAL param kind if any, and None if not. See also, Sig.index_of_var_keyword ```pycon >>> assert Sig(lambda x, *y, z: 0).index_of_var_positional == 1 >>> assert Sig(lambda x, /, y, **z: 0).index_of_var_positional == None ``` #### *property* inject_into_keyword_variadic Decorator that uses signature to source the keyword variadic of target function. See replace_kwargs_using function for more details, including examples. ```pycon >>> def apple(a, x: int, y=2, *, z=3, **extra_apple_options): ... return a + x + y + z >>> @Sig(apple).inject_into_keyword_variadic ... def sauce(a, b, c, **sauce_kwargs): ... return b * c + apple(a, **sauce_kwargs) ``` The function will works: ```pycon >>> sauce(1, 2, 3, x=4, z=5) # func still works? Should be: 1 + 4 + 2 + 5 + 2 * 3 18 ``` But the signature now doesn’t have the `**sauce_kwargs`, but more informative signature elements sourced from `apple`: ```pycon >>> Sig(sauce) ``` #### is_call_compatible_with(other_sig, , param_comparator=None) Return True if the signature is compatible with `other_sig`. Meaning that all valid ways to call the signature are valid for `other_sig`. #### kwargs_from_args_and_kwargs(args=None, kwargs=None, , apply_defaults=False, allow_partial=False, allow_excess=False, ignore_kind=False) Map arguments (args and kwargs) to the parameters of function’s signature. When you need to manage how the arguments of a function are specified, you need to take care of multiple cases depending on whether they were specified as positional arguments (`args`) or keyword arguments (`kwargs`). The `map_arguments` (and it’s sorta-inverse inverse, `mk_args_and_kwargs`) are there to help you manage this. If you could rely on the the fact that only `kwargs` were given it would reduce the complexity of your code. This is why we have the `all_pk_signature` function in `signatures.py`. We also need to have a means to make a `kwargs` only from the actual `(*args, **kwargs)` used at runtime. We have `Signature.bind` (and `bind_partial`) for that. But these methods will fail if there is extra stuff in the `kwargs`. Yet sometimes we’d like to have a `dict` that services several functions that will extract their needs from it. That’s where `Sig.map_arguments_from_variadics(*args, **kwargs)` is needed. * **Parameters:** * **args** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)) – The args the function will be called with. * **kwargs** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – The kwargs the function will be called with. * **apply_defaults** – (bool) Whether to apply signature defaults to the non-specified argument names * **allow_partial** – (bool) True iff you want to allow partial signature fulfillment. * **allow_excess** – (bool) Set to True iff you want to allow extra kwargs items to be ignored. * **ignore_kind** – (bool) Set to True iff you want to ignore the position and keyword only kinds, in order to be able to accept args and kwargs in such a way that there can be cross-over (args that are supposed to be keyword only, and kwargs that are supposed to be positional only) * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** An {param_name: arg_val, …} dict See also the sorta-inverse of this function: mk_args_and_kwargs ```pycon >>> def foo(w, /, x: float, y="YY", *, z: str = "ZZ"): ... ... >>> sig = Sig(foo) >>> assert ( ... sig.map_arguments((11, 22, "you"), dict(z="zoo")) ... == sig.map_arguments((11, 22), dict(y="you", z="zoo")) ... == {"w": 11, "x": 22, "y": "you", "z": "zoo"} ... ) ``` By default, `apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> sig.map_arguments(args=(11,), kwargs={"x": 22}) {'w': 11, 'x': 22} ``` But if you specify `apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> sig.map_arguments( ... args=(11,), kwargs={"x": 22}, apply_defaults=True ... ) {'w': 11, 'x': 22, 'y': 'YY', 'z': 'ZZ'} ``` By default, `ignore_excess=False`, so specifying kwargs that are not in the signature will lead to an exception. ```pycon >>> sig.map_arguments( ... args=(11,), kwargs={"x": 22, "not_in_sig": -1} ... ) Traceback (most recent call last): ... TypeError: got an unexpected keyword argument 'not_in_sig' ``` Specifying `allow_excess=True` will ignore such excess fields of kwargs. This is useful when you want to source several functions from a same dict. ```pycon >>> sig.map_arguments( ... args=(11,), kwargs={"x": 22, "not_in_sig": -1}, allow_excess=True ... ) {'w': 11, 'x': 22} ``` On the other side of `ignore_excess` you have `allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> sig.map_arguments(args=(), kwargs={"x": 22}) Traceback (most recent call last): ... TypeError: missing a required argument: 'w' ``` But if you specify `allow_partial=True`… ```pycon >>> sig.map_arguments( ... args=(), kwargs={"x": 22}, allow_partial=True ... ) {'x': 22} ``` That’s a lot of control (eight combinations total), but not everything is controllable here: Position only and keyword only kinds need to be respected: ```pycon >>> sig.map_arguments(args=(1, 2, 3, 4), kwargs={}) Traceback (most recent call last): ... TypeError: too many positional arguments >>> sig.map_arguments(args=(), kwargs=dict(w=1, x=2, y=3, z=4)) Traceback (most recent call last): ... TypeError:...'w'... ``` But if you want to ignore the kind of parameter, just say so: ```pycon >>> sig.map_arguments( ... args=(1, 2, 3, 4), kwargs={}, ignore_kind=True ... ) {'w': 1, 'x': 2, 'y': 3, 'z': 4} >>> sig.map_arguments( ... args=(), kwargs=dict(w=1, x=2, y=3, z=4), ignore_kind=True ... ) {'w': 1, 'x': 2, 'y': 3, 'z': 4} ``` #### map_arguments(args=None, kwargs=None, , apply_defaults=False, allow_partial=False, allow_excess=False, ignore_kind=False) Map arguments (args and kwargs) to the parameters of function’s signature. When you need to manage how the arguments of a function are specified, you need to take care of multiple cases depending on whether they were specified as positional arguments (`args`) or keyword arguments (`kwargs`). The `map_arguments` (and it’s sorta-inverse inverse, `mk_args_and_kwargs`) are there to help you manage this. If you could rely on the the fact that only `kwargs` were given it would reduce the complexity of your code. This is why we have the `all_pk_signature` function in `signatures.py`. We also need to have a means to make a `kwargs` only from the actual `(*args, **kwargs)` used at runtime. We have `Signature.bind` (and `bind_partial`) for that. But these methods will fail if there is extra stuff in the `kwargs`. Yet sometimes we’d like to have a `dict` that services several functions that will extract their needs from it. That’s where `Sig.map_arguments_from_variadics(*args, **kwargs)` is needed. * **Parameters:** * **args** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)) – The args the function will be called with. * **kwargs** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – The kwargs the function will be called with. * **apply_defaults** – (bool) Whether to apply signature defaults to the non-specified argument names * **allow_partial** – (bool) True iff you want to allow partial signature fulfillment. * **allow_excess** – (bool) Set to True iff you want to allow extra kwargs items to be ignored. * **ignore_kind** – (bool) Set to True iff you want to ignore the position and keyword only kinds, in order to be able to accept args and kwargs in such a way that there can be cross-over (args that are supposed to be keyword only, and kwargs that are supposed to be positional only) * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** An {param_name: arg_val, …} dict See also the sorta-inverse of this function: mk_args_and_kwargs ```pycon >>> def foo(w, /, x: float, y="YY", *, z: str = "ZZ"): ... ... >>> sig = Sig(foo) >>> assert ( ... sig.map_arguments((11, 22, "you"), dict(z="zoo")) ... == sig.map_arguments((11, 22), dict(y="you", z="zoo")) ... == {"w": 11, "x": 22, "y": "you", "z": "zoo"} ... ) ``` By default, `apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> sig.map_arguments(args=(11,), kwargs={"x": 22}) {'w': 11, 'x': 22} ``` But if you specify `apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> sig.map_arguments( ... args=(11,), kwargs={"x": 22}, apply_defaults=True ... ) {'w': 11, 'x': 22, 'y': 'YY', 'z': 'ZZ'} ``` By default, `ignore_excess=False`, so specifying kwargs that are not in the signature will lead to an exception. ```pycon >>> sig.map_arguments( ... args=(11,), kwargs={"x": 22, "not_in_sig": -1} ... ) Traceback (most recent call last): ... TypeError: got an unexpected keyword argument 'not_in_sig' ``` Specifying `allow_excess=True` will ignore such excess fields of kwargs. This is useful when you want to source several functions from a same dict. ```pycon >>> sig.map_arguments( ... args=(11,), kwargs={"x": 22, "not_in_sig": -1}, allow_excess=True ... ) {'w': 11, 'x': 22} ``` On the other side of `ignore_excess` you have `allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> sig.map_arguments(args=(), kwargs={"x": 22}) Traceback (most recent call last): ... TypeError: missing a required argument: 'w' ``` But if you specify `allow_partial=True`… ```pycon >>> sig.map_arguments( ... args=(), kwargs={"x": 22}, allow_partial=True ... ) {'x': 22} ``` That’s a lot of control (eight combinations total), but not everything is controllable here: Position only and keyword only kinds need to be respected: ```pycon >>> sig.map_arguments(args=(1, 2, 3, 4), kwargs={}) Traceback (most recent call last): ... TypeError: too many positional arguments >>> sig.map_arguments(args=(), kwargs=dict(w=1, x=2, y=3, z=4)) Traceback (most recent call last): ... TypeError:...'w'... ``` But if you want to ignore the kind of parameter, just say so: ```pycon >>> sig.map_arguments( ... args=(1, 2, 3, 4), kwargs={}, ignore_kind=True ... ) {'w': 1, 'x': 2, 'y': 3, 'z': 4} >>> sig.map_arguments( ... args=(), kwargs=dict(w=1, x=2, y=3, z=4), ignore_kind=True ... ) {'w': 1, 'x': 2, 'y': 3, 'z': 4} ``` #### map_arguments_from_variadics(\*args, \_apply_defaults=False, \_allow_partial=False, \_allow_excess=False, \_ignore_kind=False, \*\*kwargs) Convenience method that calls map_arguments from variadics ```pycon >>> def foo(w, /, x: float, y="YY", *, z: str = "ZZ"): ... ... >>> sig = Sig(foo) >>> assert ( ... sig.map_arguments_from_variadics(1, 2, 3, z=4) ... == sig.map_arguments_from_variadics(1, 2, y=3, z=4) ... == {"w": 1, "x": 2, "y": 3, "z": 4} ... ) ``` What about var positional and var keywords? ```pycon >>> def bar(*args, **kwargs): ... ... ... >>> Sig(bar).map_arguments_from_variadics(1, 2, y=3, z=4) {'args': (1, 2), 'kwargs': {'y': 3, 'z': 4}} ``` Note that though `w` is a position only argument, you can specify `w=11` as a keyword argument too, using `_ignore_kind=True`: ```pycon >>> Sig(foo).map_arguments_from_variadics(w=11, x=22, _ignore_kind=True) {'w': 11, 'x': 22} ``` You can use `_allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> Sig(foo).map_arguments_from_variadics(x=3, y=2) Traceback (most recent call last): ... TypeError: missing a required argument: 'w' ``` But if you specify `_allow_partial=True`… ```pycon >>> Sig(foo).map_arguments_from_variadics(x=3, y=2, _allow_partial=True) {'x': 3, 'y': 2} ``` By default, `_apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> Sig(foo).map_arguments_from_variadics(4, x=3, y=2) {'w': 4, 'x': 3, 'y': 2} ``` But if you specify `_apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> Sig(foo).map_arguments_from_variadics(4, x=3, y=2, _apply_defaults=True) {'w': 4, 'x': 3, 'y': 2, 'z': 'ZZ'} ``` #### merge_with_sig(sig, ch_to_all_pk=False, , default_conflict_method='strict') Return a signature obtained by merging self signature with another signature. Insofar as it can, given the kind precedence rules, the arguments of self will appear first. * **Parameters:** * **sig** (`Union`[[`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The signature to merge with. * **ch_to_all_pk** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to change all kinds of both signatures to PK ( POSITIONAL_OR_KEYWORD) * **Returns:** ```pycon >>> def func(a=None, *, b=1, c=2): ... ... ... >>> >>> s = Sig(func) >>> s ``` Observe where the new arguments `d` and `e` are placed, according to whether they have defaults and what their kind is: ```pycon >>> s.merge_with_sig(["d", "e"]) >>> s.merge_with_sig(["d", ("e", 4)]) >>> s.merge_with_sig(["d", dict(name="e", kind=KO, default=4)]) >>> s.merge_with_sig( ... [dict(name="d", kind=KO), dict(name="e", kind=KO, default=4)] ... ) ``` If the kind of the params is not important, but order is, you can specify `ch_to_all_pk=True`: ```pycon >>> s.merge_with_sig(["d", "e"], ch_to_all_pk=True) >>> s.merge_with_sig([("d", 3), ("e", 4)], ch_to_all_pk=True) ``` #### mk_args_and_kwargs(arguments, , apply_defaults=False, allow_partial=False, allow_excess=False, ignore_kind=False, args_limit=0) Extract args and kwargs such that `func(*args, **kwargs)` can be called, where func has instance’s signature. * **Parameters:** * **arguments** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – The {param_name: arg_val,…} dict to process * **args_limit** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – How “far” in the params should args (positional arguments) be searched for. - args_limit==0: Take the minimum number possible of args (positional arguments). Only those that are position only or before a var-positional. - args_limit is None: Take the maximum number of args (positional arguments). The only kwargs (keyword arguments) you should have are keyword-only and var-keyword arguments. - args_limit positive integer: Take the args_limit first argument names (of signature) as args, and the rest as kwargs. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ```pycon >>> def foo(w, /, x: float, y=1, *, z: int = 1): ... return ((w + x) * y) ** z >>> foo_sig = Sig(foo) >>> args, kwargs = foo_sig.mk_args_and_kwargs( ... dict(w=4, x=3, y=2, z=1) ... ) >>> assert (args, kwargs) == ((4,), {"x": 3, "y": 2, "z": 1}) >>> assert foo(*args, **kwargs) == foo(4, 3, 2, z=1) == 14 ``` What about variadics? ```pycon >>> def bar(a, /, b, *args, c=2, **kwargs): ... pass >>> Sig(bar).mk_args_and_kwargs( ... dict(a=1, b=2, args=(3,4), c=5, kwargs=dict(d=6, e=7)) ... ) ((1, 2, 3, 4), {'c': 5, 'd': 6, 'e': 7}) ``` You can also give the arguments in a different order: ```pycon >>> Sig(bar).mk_args_and_kwargs( ... dict(args=(3,4), kwargs=dict(d=6, e=7), b=2, c=5, a=1) ... ) ((1, 2, 3, 4), {'c': 5, 'd': 6, 'e': 7}) ``` The `args_limit` begs explanation. Consider the signature of `def foo(w, /, x: float, y=1, *, z: int = 1): ...` for instance. We could call the function with the following (args, kwargs) pairs: - ((1,), {‘x’: 2, ‘y’: 3, ‘z’: 4}) - ((1, 2), {‘y’: 3, ‘z’: 4}) - ((1, 2, 3), {‘z’: 4}) The two other combinations (empty args or empty kwargs) are not valid because of the / and \* constraints. But when asked for an (args, kwargs) pair, which of the three valid options should be returned? This is what the `args_limit` argument controls. If `args_limit == 0`, the least args (positional arguments) will be returned. It’s the default. ```pycon >>> arguments = dict(w=4, x=3, y=2, z=1) >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=0) ((4,), {'x': 3, 'y': 2, 'z': 1}) ``` If `args_limit is None`, the least kwargs (keyword arguments) will be returned. ```pycon >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=None) ((4, 3, 2), {'z': 1}) ``` If `args_limit` is a positive integer, the first `[args_limit]` arguments will be returned (not checking at all if this is valid!). ```pycon >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=1) ((4,), {'x': 3, 'y': 2, 'z': 1}) >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=2) ((4, 3), {'y': 2, 'z': 1}) >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=3) ((4, 3, 2), {'z': 1}) ``` Note that if you specify `args_limit` to be greater than the maximum of positional arguments, it behaves as if `args_limit` was `None`: ```pycon >>> foo_sig.mk_args_and_kwargs(arguments, args_limit=4) ((4, 3, 2), {'z': 1}) ``` Note that ‘args_limit’’s behavior is consistent with list behvior in the sense that: ```pycon >>> args = (0, 1, 2, 3) >>> args[:0] () >>> args[:None] (0, 1, 2, 3) >>> args[2] 2 ``` If variable positional arguments are present, `args_limit` is ignored and all positional arguments are returned as args. ```pycon >>> Sig(bar).mk_args_and_kwargs( ... dict(a=1, b=2, args=(3,4), c=5, kwargs=dict(d=6, e=7)), ... args_limit=1 ... ) ((1, 2, 3, 4), {'c': 5, 'd': 6, 'e': 7}) ``` By default, only the arguments that were given in the `arguments` input will be returned in the (args, kwargs) output. If you also want to get those that have defaults (according to signature), you need to specify it with the `apply_defaults=True` argument. ```pycon >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3)) ((4,), {'x': 3}) >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3), apply_defaults=True) ((4,), {'x': 3, 'y': 1, 'z': 1}) ``` By default, all required arguments must be given. Not doing so will lead to a `TypeError`. If you want to process your arguments anyway, specify `allow_partial=True`. ```pycon >>> foo_sig.mk_args_and_kwargs(dict(w=4)) Traceback (most recent call last): ... TypeError: missing a required argument: 'x' >>> foo_sig.mk_args_and_kwargs(dict(w=4), allow_partial=True) ((4,), {}) ``` Specifying argument names that are not recognized by the signature will lead to a `TypeError`. If you want to avoid this (and just take from the input `kwargs` what ever you can), specify this with `allow_excess=True`. ```pycon >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3, extra='stuff')) Traceback (most recent call last): ... TypeError: Got unexpected keyword arguments: extra >>> foo_sig.mk_args_and_kwargs(dict(w=4, x=3, extra='stuff'), ... allow_excess=True) ((4,), {'x': 3}) ``` See `map_arguments` (namely for the description of the arguments). #### modified(\_allow_reordering=False, \*\*changes_for_name) Returns a modified (new) signature object. #### NOTE This function doesn’t modify the signature, but creates a modified copy of the signature. IMPORTANT WARNING: This is an advanced feature. Avoid wrapping a function with a modified signature, as this may not have the intended effect. ```pycon >>> def foo(pka, *vpa, koa, **vka): ... >>> sig = Sig(foo) >>> sig >>> assert sig.kinds['pka'] == PK ``` Let’s make a signature that is the same as sig, except that > - `poa` is given a PO (POSITIONAL_ONLY) kind insteadk of PK > - `koa` is given a default of None > - the signature is given a return_annotation of str ```pycon >>> new_sig = sig.modified( ... pka={'kind': PO}, ... koa={'default': None}, ... return_annotation=str ... ) >>> new_sig str> >>> assert new_sig.kinds['pka'] == PO # now pos is of the PO kind! ``` Here’s an example of changing signature parameters in bulk. Here we change all kinds to be the friendly PK kind. ```pycon >>> sig.modified(**{name: {'kind': PK} for name in sig.names}) ``` Repetition of the above: This gives you a signature with all PK kinds. If you wrap a function with it, it will look like it has all PK kinds. But that doesn’t mean you can actually use thenm as such. You’ll need to modify (decorate further) your function further to reflect its new signature. On the other hand, if you decorate a function with a sig that adds or modifies defaults, these defaults will actually be used (unlike with `functools.wraps`). #### *property* n_required The number of required arguments. A required argument is one that doesn’t have a default, nor is VAR_POSITIONAL (`*args`) or VAR_KEYWORD (`**kwargs`). #### NOTE Sometimes a minimum number of arguments in VAR_POSITIONAL and VAR_KEYWORD are in fact required, but we can’t see this from the signature, so we can’t tell you about that! You do the math. ```pycon >>> f = lambda a00, /, a11, a12, *a23, a34, a35=1, a36='two', **a47: None >>> Sig(f).n_required 4 ``` #### names_for_kind(kind) Get the arg names tuple for a given kind. Note, if you need to do this several times, or for several kinds, use `names_of_kind` property (a tuple) instead: It groups all names of kinds once, and caches the result. #### pair_with(other_sig) Get an object that pairs with another signature for comparison, merging, etc. See `SigPair` for more details. * **Return type:** [`SigPair`](_autosummary/dol.signatures.html.md#dol.signatures.SigPair) #### *property* params Just list(self.parameters.values()), because that’s often what we want. Why a Sig.params property when we already have a Sig.parameters property? Well, as much as is boggles my mind, it so happens that the Signature.parameters is a name->Parameter mapping, but the Signature argument `parameters`, though baring the same name, is expected to be a list of Parameter instances. So Sig.params is there to restore semantic consistence sanity. #### replace_kwargs_using() Decorator that replaces the variadic keyword argument of the target function using the `sig`, the signature of a source function. It essentially injects the difference between `sig` and the target function’s signature into the target function’s signature. That is, it replaces the variadic keyword argument (a.k.a. “kwargs”) with those parameters that are in `sig` but not in the target function’s signature. This is meant to be used when a `targ_func` (the function you’ll apply the decorator to) has a variadict keyword argument that is just used to forward “extra” arguments to another function, and you want to make sure that the signature of the `targ_func` is consistent with the `sig` signature. (Also, you don’t want to copy the signatures around manually.) In the following, `sauce` (the target function) has a variadic keyword argument, `sauce_kwargs`, that is used to forward extra arguments to `apple` (the source function). ```pycon >>> def apple(a, x: int, y=2, *, z=3, **extra_apple_options): ... return a + x + y + z >>> @replace_kwargs_using(apple) ... def sauce(a, b, c, **sauce_kwargs): ... return b * c + apple(a, **sauce_kwargs) ``` The function will works: ```pycon >>> sauce(1, 2, 3, x=4, z=5) # func still works? Should be: 1 + 4 + 2 + 5 + 2 * 3 18 ``` But the signature now doesn’t have the `**sauce_kwargs`, but more informative signature elements sourced from `apple`: ```pycon >>> Sig(sauce) ``` One thing to note is that the order of the arguments in the signature of `apple` may change to accomodate for the python parameter order rules (see [https://docs.python.org/3/reference/compound_stmts.html#function-definitions](https://docs.python.org/3/reference/compound_stmts.html#function-definitions)). The new order will try to conserve the order of the original arguments of `sauce` in-so-far as it doesn’t violate the python parameter order rules, though. See examples below: ```pycon >>> @Sig.replace_kwargs_using(apple) ... def sauce(a, b=2, c=3, **sauce_kwargs): ... return b * c + apple(a, **sauce_kwargs) >>> Sig(sauce) ``` ```pycon >>> @Sig.replace_kwargs_using(apple) ... def sauce(a=1, b=2, c=3, **sauce_kwargs): ... return b * c + apple(a, **sauce_kwargs) >>> Sig(sauce) ``` #### *property* required_names A tuple of required names, preserving the original signature order. A required name is that must be given in a function call, that is, the name of a paramater that doesn’t have a default, and is not a variadic. That lost one is a frequent gotcha, so oo not fall in that gotcha that easily, we provide a property that contains what we need. ```pycon >>> f = lambda a00, /, a11, a12, *a23, a34, a35=1, a36='two', **a47: None >>> Sig(f).required_names ('a00', 'a11', 'a12', 'a34') ``` #### *classmethod* sig_or_default(obj, default_signature=) Returns a Sig instance, or a default signature if there was a ValueError trying to construct it. For example, `time.time` doesn’t have a signature ```pycon >>> import time >>> has_signature(time.time) False ``` But we can tell `Sig` to give it the default one: ```pycon >>> str(Sig.sig_or_default(time.time)) '(*no_sig_args, **no_sig_kwargs)' ``` That’s the default signature, which should work for most purposes. You can also specify what the default should be though. ```pycon >>> fake_signature = Sig(lambda *time_takes_no_arguments: ...) >>> str(Sig.sig_or_default(time.time, fake_signature)) '(*time_takes_no_arguments)' ``` Careful though. If you assign a signature to a function that is not aligned with that actually functioning of the function, bad things will happen. In this case, the actual signature of time is the empty signature: ```pycon >>> str(Sig.sig_or_default(time.time, Sig(lambda: ...))) '()' ``` #### *classmethod* sig_or_none(obj) Returns a Sig instance, or None if there was a ValueError trying to construct it. One use case is to be able to tell if an object has a signature or not. ```pycon >>> robust_has_signature = lambda obj: bool(Sig.sig_or_none(obj)) >>> robust_has_signature(robust_has_signature) # an easy case True >>> robust_has_signature( ... Sig ... ) # another easy one: This time, a type/class (which is callable, yes) True ``` But here’s where it get’s interesting. `print`, a builtin, doesn’t have a signature through inspect.signature. ```pycon >>> has_signature(print) False ``` But we do get one with robust_has_signature ```pycon >>> robust_has_signature(print) True ``` #### sort_params() Returns a signature with the parameters sorted by kind and default presence. #### source_args_and_kwargs(\*args, \_ignore_kind=True, \_allow_partial=False, \_apply_defaults=False, \_args_limit=0, \*\*kwargs) Source the (args, kwargs) for the signature instance, ignoring excess arguments. ```pycon >>> def foo(w, /, x: float, y=2, *, z: int = 1): ... return w + x * y ** z >>> args, kwargs = Sig(foo).source_args_and_kwargs( ... 4, x=3, y=2, extra="keywords", are="ignored" ... ) >>> args, kwargs ((4,), {'x': 3, 'y': 2}) ``` The difference with source_arguments is that here the output is ready to be called by the function whose signature we have, since the position-only arguments will be returned as args. ```pycon >>> foo(*args, **kwargs) 10 ``` Note that though `w` is a position only argument, you can specify `w=4` as a keyword argument too (by default): ```pycon >>> args, kwargs = Sig(foo).source_args_and_kwargs( ... w=4, x=3, y=2, extra="keywords", are="ignored" ... ) >>> assert (args, kwargs) == ((4,), {"x": 3, "y": 2}) ``` If you don’t want to allow that, you can say `_ignore_kind=False` ```pycon >>> Sig(foo).source_args_and_kwargs( ... w=4, x=3, y=2, extra="keywords", are="ignored", _ignore_kind=False ... ) Traceback (most recent call last): ... TypeError: ...'w'... ``` You can use `_allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> Sig(foo).source_args_and_kwargs(x=3, y=2, extra="keywords", are="ignored") Traceback (most recent call last): ... TypeError:...'w'... ``` But if you specify `_allow_partial=True`… ```pycon >>> args, kwargs = Sig(foo).source_args_and_kwargs( ... x=3, y=2, extra="keywords", are="ignored", _allow_partial=True ... ) >>> (args, kwargs) == ((), {"x": 3, "y": 2}) True ``` By default, `_apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> args, kwargs = Sig(foo).source_args_and_kwargs( ... 4, x=3, y=2, extra="keywords", are="ignored" ... ) >>> (args, kwargs) == ((4,), {"x": 3, "y": 2}) True ``` But if you specify `_apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> args, kwargs = Sig(foo).source_args_and_kwargs( ... 4, x=3, y=2, extra="keywords", are="ignored", _apply_defaults=True ... ) >>> (args, kwargs) == ((4,), {"x": 3, "y": 2, "z": 1}) True ``` #### source_arguments(\*args, \_apply_defaults=False, \_allow_partial=False, \_ignore_kind=True, \*\*kwargs) Source the arguments for the signature instance, ignoring excess arguments. ```pycon >>> def foo(w, /, x: float, y="YY", *, z: str = "ZZ"): ... ... >>> Sig(foo).source_arguments(11, x=22, extra="keywords", are="ignored") {'w': 11, 'x': 22} ``` Note that though `w` is a position only argument, you can specify `w=11` as a keyword argument too (by default): ```pycon >>> Sig(foo).source_arguments(w=11, x=22, extra="keywords", are="ignored") {'w': 11, 'x': 22} ``` If you don’t want to allow that, you can say `_ignore_kind=False` ```pycon >>> Sig(foo).source_arguments( ... w=11, x=22, extra="keywords", are="ignored", _ignore_kind=False ... ) Traceback (most recent call last): ... TypeError: ...'w'... ``` You can use `_allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> Sig(foo).source_arguments(x=3, y=2, extra="keywords", are="ignored") Traceback (most recent call last): ... TypeError: ...'w'... ``` But if you specify `_allow_partial=True`… ```pycon >>> Sig(foo).source_arguments( ... x=3, y=2, extra="keywords", are="ignored", _allow_partial=True ... ) {'x': 3, 'y': 2} ``` By default, `_apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> Sig(foo).source_arguments(4, x=3, y=2, extra="keywords", are="ignored") {'w': 4, 'x': 3, 'y': 2} ``` But if you specify `_apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> Sig(foo).source_arguments( ... 4, x=3, y=2, extra="keywords", are="ignored", _apply_defaults=True ... ) {'w': 4, 'x': 3, 'y': 2, 'z': 'ZZ'} ``` #### source_kwargs(\*args, \_apply_defaults=False, \_allow_partial=False, \_ignore_kind=True, \*\*kwargs) Source the arguments for the signature instance, ignoring excess arguments. ```pycon >>> def foo(w, /, x: float, y="YY", *, z: str = "ZZ"): ... ... >>> Sig(foo).source_arguments(11, x=22, extra="keywords", are="ignored") {'w': 11, 'x': 22} ``` Note that though `w` is a position only argument, you can specify `w=11` as a keyword argument too (by default): ```pycon >>> Sig(foo).source_arguments(w=11, x=22, extra="keywords", are="ignored") {'w': 11, 'x': 22} ``` If you don’t want to allow that, you can say `_ignore_kind=False` ```pycon >>> Sig(foo).source_arguments( ... w=11, x=22, extra="keywords", are="ignored", _ignore_kind=False ... ) Traceback (most recent call last): ... TypeError: ...'w'... ``` You can use `_allow_partial` that will allow you, if set to `True`, to underspecify the params of a function (in view of being completed later). ```pycon >>> Sig(foo).source_arguments(x=3, y=2, extra="keywords", are="ignored") Traceback (most recent call last): ... TypeError: ...'w'... ``` But if you specify `_allow_partial=True`… ```pycon >>> Sig(foo).source_arguments( ... x=3, y=2, extra="keywords", are="ignored", _allow_partial=True ... ) {'x': 3, 'y': 2} ``` By default, `_apply_defaults=False`, which will lead to only get those arguments you input. ```pycon >>> Sig(foo).source_arguments(4, x=3, y=2, extra="keywords", are="ignored") {'w': 4, 'x': 3, 'y': 2} ``` But if you specify `_apply_defaults=True` non-specified non-require arguments will be returned with their defaults: ```pycon >>> Sig(foo).source_arguments( ... 4, x=3, y=2, extra="keywords", are="ignored", _apply_defaults=True ... ) {'w': 4, 'x': 3, 'y': 2, 'z': 'ZZ'} ``` #### to_signature_kwargs() The dict of keyword arguments to make this signature instance. ```pycon >>> def f(w, /, x: float = 2, y=1, *, z: int = 0) -> float: ... ... >>> Sig(f).to_signature_kwargs() {'parameters': [, , , ], 'return_annotation': } ``` Note that this does NOT return: ```python {'parameters': self.parameters, 'return_annotation': self.return_annotation} ``` which would not actually work as keyword arguments of `Signature`. Yeah, I know. Don’t ask me, ask the authors of `Signature`! Instead, `parammeters` will be `list(self.parameters.values())`, which does work. #### to_simple_signature() A builtin `inspect.Signature` instance equivalent (i.e. without the extra properties and methods) ```pycon >>> def f(w, /, x: float = 2, y=1, *, z: int = 0): ... ... >>> Sig(f).to_simple_signature() ``` #### *property* with_defaults Sub-signature containing only “not required” (i.e. with defaults) parameters. ```pycon >>> list(Sig(lambda *args, a, b, x=1, y=1, **kwargs: ...).with_defaults) ['args', 'x', 'y', 'kwargs'] ``` #### *property* without_defaults Sub-signature containing only “required” (i.e. without defaults) parameters. ```pycon >>> list(Sig(lambda *args, a, b, x=1, y=1, **kwargs: ...).without_defaults) ['a', 'b'] ``` #### wrap(func, ignore_incompatible_signatures=True, , copy_function=False) Gives the input function the signature. This is similar to the `functools.wraps` function, but parametrized by a signature (not a callable). Also, where as both write to the input func’s `__signature__` attribute, here we also write to - `__defaults__` and `__kwdefaults__`, extracting these from `__signature__` (functools.wraps doesn’t do that at the time of writing this (see [https://github.com/python/cpython/pull/21379](https://github.com/python/cpython/pull/21379))). - `__annotations__` (also extracted from `__signature__`) - does not write to `__module__`, `__name__`, `__qualname__`, `__doc__` (because again, we’re basinig the injecton on a signature, not a function, so we have no name, doc, etc…) #### WARNING The fact that you’ve modified the signature of your function doesn’t mean that the decorated function will work as expected (or even work at all). See below for examples. ```pycon >>> def f(w, /, x: float = 1, y=2, z: int = 3): ... return w + x * y ** z >>> f(0, 1) # 0 + 1 * 2 ** 3 8 >>> f.__defaults__ (1, 2, 3) >>> assert 8 == f(0) == f(0, 1) == f(0, 1, 2) == f(0, 1, 2, 3) ``` Now let’s create a very similar function to f, but where: - w is not position-only - x annot is int instead of float, and doesn’t have a default - z’s default changes to 10 ```pycon >>> def g(w, x: int, y=2, z: int = 10): ... return w + x * y ** z >>> s = Sig(g) >>> f = s.wrap(f) >>> import inspect >>> inspect.signature(f) # see that >>> # But (unlike with functools.wraps) here we get __defaults__ and __kwdefault__ >>> f.__defaults__ # see that x has no more default & z's default is now 10 (2, 10) >>> f( ... 0, 1 ... ) # see that now we get a different output because using different defaults 1024 ``` Remember that you are modifying the signature, not the function itself. Signature changes in defaults will indeed change the function’s behavior. But changes in name or kind will only be reflected in the signature, and misalignment with the wrapped function will lead to unexpected results. ```pycon >>> def f(w, /, x: float = 1, y=2, *, z: int = 3): ... return w + x * y ** z >>> f(0) # 0 + 1 * 2 ** 3 8 >>> f(0, 1, 2, 3) # error expected! Traceback (most recent call last): ... TypeError: f() takes from 1 to 3 positional arguments but 4 were given ``` But if you try to remove the argument kind constraint by just changing the signature, you’ll fail. ```pycon >>> def g(w, x: float = 1, y=2, z: int = 3): ... return w + x * y ** z >>> f = Sig(g).wrap(f) >>> f(0) Traceback (most recent call last): ... TypeError: f() missing 1 required keyword-only argument: 'z' >>> f(0, 1, 2, 3) Traceback (most recent call last): ... TypeError: f() takes from 0 to 3 positional arguments but 4 were given ``` ### *class* dol.signatures.SigPair(sig1, sig2) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Class that operates on a pair of signatures. For example, offers methods to compare two signatures in various ways. * **Parameters:** * **sig1** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`Sig`](_autosummary/dol.signatures.html.md#dol.signatures.Sig)) – First signature or signature-able object. * **sig2** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`Sig`](_autosummary/dol.signatures.html.md#dol.signatures.Sig)) – Second signature or signature-able object. ```pycon >>> from pprint import pprint >>> def three(a, b: int, c=3): ... >>> def little(a, *, b=2, d=4) -> int: ... >>> def pigs(a, b) -> int: ... >>> sig_pair = SigPair(three, little) >>> >>> sig_pair.shared_names ['a', 'b'] >>> sig_pair.names_missing_in_sig1 ['d'] >>> sig_pair.names_missing_in_sig2 ['c'] >>> sig_pair.param_comparison() False >>> pprint(sig_pair.diff()) {'names_missing_in_sig1': ['d'], 'names_missing_in_sig2': ['c'], 'param_differences': {'b': {'annotation': (, ), 'default': (, 2), 'kind': (<_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <_ParameterKind.KEYWORD_ONLY: 3>)}}, 'return_annotation': (, )} ``` Call compatibility says that any arguments leading to a valid call to a function having the first signature, will also lead to a valid call to a function having the second signature. This is not the case for the signatures of `three` and `little`: ```pycon >>> sig_pair.are_call_compatible() False ``` But we don’t need to have equal signatures to have call compatibility. For example, ```pycon >>> SigPair(three, lambda a, b=2, c=30: None).are_call_compatible() True ``` Note that call-compatibility is not symmetric. For example, `pigs` is call compatible with `three`, since any arguments that are valid for `pigs` are valid for `three`: ```pycon >>> SigPair(pigs, three).are_call_compatible() True ``` But `three` is not call-compatible with `pigs` since `three` requires could include a `c` argument, which `pigs` would choke on. ```pycon >>> SigPair(three, pigs).are_call_compatible() False ``` #### are_call_compatible(param_comparator=None) Check if the signatures are call-compatible. Returns True if sig1 can be used to call sig2 or vice versa. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ```pycon >>> sig1 = Sig(lambda a, b, c=3: None) >>> sig2 = Sig(lambda a, b: None) >>> comp = SigPair(sig1, sig2) >>> comp.are_call_compatible() False ``` ```pycon >>> comp = SigPair(sig2, sig1) >>> comp.are_call_compatible() True ``` #### diff() Get a dictionary of differences between the two signatures. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ```pycon >>> from pprint import pprint >>> def three(a, b: int, c=3): ... >>> def little(a, *, b=2, d=4) -> int: ... >>> def pigs(a, b: int = 2) -> int: ... >>> pprint(SigPair(three, little).diff()) {'names_missing_in_sig1': ['d'], 'names_missing_in_sig2': ['c'], 'param_differences': {'b': {'annotation': (, ), 'default': (, 2), 'kind': (<_ParameterKind.POSITIONAL_OR_KEYWORD: 1>, <_ParameterKind.KEYWORD_ONLY: 3>)}}, 'return_annotation': (, )} >>> pprint(SigPair(three, pigs).diff()) {'names_missing_in_sig2': ['c'], 'param_differences': {'b': {'default': (, 2)}}, 'return_annotation': (, )} >>> pprint(SigPair(three, three).diff()) {} ``` #### diff_str() Get a string representation of the differences between the two signatures. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) #### *property* names_missing_in_sig1 List of names that are in the sig2 signature but not in sig1. ```pycon >>> sig1 = Sig(lambda a, b, c: None) >>> sig2 = Sig(lambda b, c, d: None) >>> comp = SigPair(sig1, sig2) >>> comp.names_missing_in_sig1 ['d'] ``` #### *property* names_missing_in_sig2 List of names that are in the sig1 signature but not in sig2. ```pycon >>> sig1 = Sig(lambda a, b, c: None) >>> sig2 = Sig(lambda b, c, d: None) >>> comp = SigPair(sig1, sig2) >>> comp.names_missing_in_sig2 ['a'] ``` #### param_comparison(comparator=, aggregation=) Compare parameters between the two signatures using the provided comparator function. * **Parameters:** * **comparator** – A function to compare two parameters. * **aggregation** – A function to aggregate the results of the comparisons. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) * **Returns:** Boolean result of the aggregated comparisons. ```pycon >>> sig1 = Sig('(a, b: int, c=3)') >>> sig2 = Sig('(a, *, b=2, d=4)') >>> comp = SigPair(sig1, sig2) >>> comp.param_comparison() False ``` #### param_differences() Get a dictionary of parameter differences between the two signatures. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** A dict containing differences for each shared param that has any. ```pycon >>> sig1 = Sig('(a, b: int, c=3)') >>> sig2 = Sig('(a, *, b=2, d=4)') >>> comp = SigPair(sig1, sig2) >>> result = comp.param_differences() >>> expected = { ... 'b': { ... 'kind': (Parameter.POSITIONAL_OR_KEYWORD, Parameter.KEYWORD_ONLY), ... 'default': (Parameter.empty, 2), ... 'annotation': (int, Parameter.empty), ... } ... } >>> result == expected True ``` #### *property* shared_names List of names that are common to both signatures, in the order of sig1. ```pycon >>> sig1 = Sig(lambda a, b, c: None) >>> sig2 = Sig(lambda b, c, d: None) >>> comp = SigPair(sig1, sig2) >>> comp.shared_names ['b', 'c'] ``` ### dol.signatures.all_pk_signature(callable_or_signature) Changes all (non-variadic) arguments to be of the PK (POSITION_OR_KEYWORD) kind. Wrapping a function with the resulting signature doesn’t make that function callable with PK kinds in itself. It just gives it a signature without position and keyword ONLY kinds. It should be used to wrap such a function that actually carries out the implementation though! ```pycon >>> def foo(w, /, x: float, y=1, *, z: int = 1, **kwargs): ... ... >>> def bar(*args, **kwargs): ... ... ... >>> from inspect import signature >>> new_foo = all_pk_signature(foo) >>> Sig(new_foo) >>> all_pk_signature(signature(foo)) ``` But note that the variadic arguments `*args` and `**kwargs` remain variadic: ```pycon >>> all_pk_signature(signature(bar)) ``` It works with `Sig` too (since Sig is a Signature), and maintains it’s other attributes (like name). ```pycon >>> sig = all_pk_signature(Sig(bar)) >>> sig >>> sig.name 'bar' ``` #### SEE ALSO `i2.signatures.kind_forgiving_func` ### dol.signatures.assure_params(obj=None) Get an interable of Parameter instances from an object. * **Parameters:** **obj** (`Union`[[`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) * **Returns:** From a callable: ```pycon >>> def f(w, /, x: float = 1, y=1, *, z: int = 1): ... ... >>> ensure_params(f) [, , , ] ``` From an iterable of strings, dicts, or tuples ```pycon >>> ensure_params( ... [ ... "xyz", ... ( ... "b", ... Parameter.empty, ... int, ... ), # if you want an annotation without a default use Parameter.empty ... ( ... "c", ... 2, ... ), # if you just want a default, make it the second element of your tup ... dict(name="d", kind=Parameter.VAR_KEYWORD), ... ] ... ) # all kinds are by default PK: Use dict to specify otherwise. [, , , ] ``` If no input is given, an empty list is returned. ```pycon >>> ensure_params() # equivalent to ensure_params(None) [] ``` ### dol.signatures.call_forgivingly(func, \*args, \*\*kwargs) Call function on given args and kwargs, but only taking what the function needs (not choking if they’re extras variables) #### TIP If you into trouble because your kwargs has a ‘func’ key, (which would then clash with the `func` param of call_forgivingly), then use `_call_forgivingly` instead, specifying args and kwargs as tuple and dict. ```pycon >>> def foo(a, b: int = 0, c=None) -> int: ... return "foo", (a, b, c) >>> call_forgivingly( ... foo, # the function you want to call ... "input for a", # meant for a -- the first (and only) argument foo requires ... c=42, # skiping b and giving c a non-default value ... intruder="argument", # but wait, this argument name doesn't exist! Oh no! ... ) # well, as it happens, nothing bad -- the intruder argument is just ignored ('foo', ('input for a', 0, 42)) ``` An example of what happens when variadic kinds are involved: ```pycon >>> def bar(x, *args1, y=1, **kwargs1): ... return x, args1, y, kwargs1 >>> call_forgivingly(bar, 1, 2, 3, y=4, z=5) (1, (2, 3), 4, {'z': 5}) ``` ### dol.signatures.call_somewhat_forgivingly(func, args, kwargs, enforce_sig=None) Call function on given args and kwargs, but with controllable argument leniency. By default, the function will only pick from args and kwargs what matches it’s signature, ignoring anything else in args and kwargs. But the real use of `call_somewhat_forgivingly` kicks in when you specify a `enforce_sig`: A signature (or any object that can be resolved into a signature through `Sig(enforce_sig)`) that will be used to bind the inputs, thus validating them against the `enforce_sig` signature (including extra arguments, defaults, etc.). `call_somewhat_forgivingly` helps you do this kind of thing systematically. ```pycon >>> f = lambda a: a * 11 >>> assert call_somewhat_forgivingly(f, (2,), {}) == f(2) ``` In the above, we have no `enforce_sig`. The real use of call_somewhat_forgivingly is when we ask it to enforce a signature. Let’s do this by specifying a function (no need for it to do anything: Only the signature is used. ```pycon >>> g = lambda a, b=None: ... ``` Calling `f` on it’s normal set of inputs (one input in this case) gives you the same thing as `f`: ```pycon >>> assert call_somewhat_forgivingly(f, (2,), {}, enforce_sig=g) == f(2) >>> assert call_somewhat_forgivingly(f, (), {'a': 2}, enforce_sig=g) == f(2) ``` If you call with an extra positional argument, it will just be ignored. ```pycon >>> assert call_somewhat_forgivingly(f, (2, 'ignored'), {}, enforce_sig=g) == f(2) ``` If you call with a `b` keyword-argument (which matches `g`’s signature, it will also be ignored. ```pycon >>> assert call_somewhat_forgivingly( ... f, (2,), {'b': 'ignored'}, enforce_sig=g ... ) == f(2) >>> assert call_somewhat_forgivingly( ... f, (), {'a': 2, 'b': 'ignored'}, enforce_sig=g ... ) == f(2) ``` But if you call with three positional arguments (one more than g allows), or call with a keyword argument that is not in `g`’s signature, it will raise a `TypeError`: ```pycon >>> call_somewhat_forgivingly(f, ... (2, 'ignored', 'does_not_fit_g_signature_anymore'), {}, enforce_sig=g ... ) Traceback (most recent call last): ... TypeError: too many positional arguments >>> call_somewhat_forgivingly(f, ... (2,), {'this_argname': 'is not in g'}, enforce_sig=g ... ) Traceback (most recent call last): ... TypeError: got an unexpected keyword argument 'this_argname' ``` ### dol.signatures.ch_func_to_all_pk(func) Returns a decorated function where all arguments are of the PK kind. (PK: Positional_or_keyword) * **Parameters:** **func** – A callable * **Returns:** ```pycon >>> def f(a, /, b, *, c=None, **kwargs): ... return a + b * c ... >>> print(Sig(f)) (a, /, b, *, c=None, **kwargs) >>> ff = ch_func_to_all_pk(f) >>> print(Sig(ff)) (a, b, c=None, **kwargs) >>> ff(1, 2, 3) 7 >>> >>> def g(x, y=1, *args, **kwargs): ... ... ... >>> print(Sig(g)) (x, y=1, *args, **kwargs) >>> gg = ch_func_to_all_pk(g) >>> print(Sig(gg)) (x, y=1, args=(), **kwargs) ``` ### dol.signatures.ch_signature_to_all_pk(callable_or_signature) Changes all (non-variadic) arguments to be of the PK (POSITION_OR_KEYWORD) kind. Wrapping a function with the resulting signature doesn’t make that function callable with PK kinds in itself. It just gives it a signature without position and keyword ONLY kinds. It should be used to wrap such a function that actually carries out the implementation though! ```pycon >>> def foo(w, /, x: float, y=1, *, z: int = 1, **kwargs): ... ... >>> def bar(*args, **kwargs): ... ... ... >>> from inspect import signature >>> new_foo = all_pk_signature(foo) >>> Sig(new_foo) >>> all_pk_signature(signature(foo)) ``` But note that the variadic arguments `*args` and `**kwargs` remain variadic: ```pycon >>> all_pk_signature(signature(bar)) ``` It works with `Sig` too (since Sig is a Signature), and maintains it’s other attributes (like name). ```pycon >>> sig = all_pk_signature(Sig(bar)) >>> sig >>> sig.name 'bar' ``` #### SEE ALSO `i2.signatures.kind_forgiving_func` ### dol.signatures.ch_variadics_to_non_variadic_kind(func, , ch_variadic_keyword_to_keyword=True) A decorator that will change a VAR_POSITIONAL (`*args`) argument to a tuple (args) argument of the same name. Essentially, given a `func(a, *b, c, **d)` function want to get a `new_func(a, b=(), c=None, d={})` that has the same functionality (in fact, calls the original `func` function behind the scenes), but without where the variadic arguments `*b` and `**d` are replaced with a `b` expecting an iterable (e.g. tuple/list) and `d` expecting a `dict` to contain the desired inputs. Besides this, the decorator tries to be as conservative as possible, making only the minimum changes needed to meet the goal of getting to a variadic-less interface. When it doubt, and error will be raised. ```pycon >>> def foo(a, *args, bar, **kwargs): ... return f"{a=}, {args=}, {bar=}, {kwargs=}" >>> assert str(Sig(foo)) == '(a, *args, bar, **kwargs)' >>> wfoo = ch_variadics_to_non_variadic_kind(foo) >>> str(Sig(wfoo)) '(a, args=(), *, bar, kwargs={})' ``` And now to do this: ```pycon >>> foo(1, 2, 3, bar=4, hello="world") "a=1, args=(2, 3), bar=4, kwargs={'hello': 'world'}" ``` We can do it like this instead: ```pycon >>> wfoo(1, (2, 3), bar=4, kwargs=dict(hello="world")) "a=1, args=(2, 3), bar=4, kwargs={'hello': 'world'}" ``` Note, the outputs are the same. It’s just the way we call our function that has changed. ```pycon >>> assert wfoo(1, (2, 3), bar=4, kwargs=dict(hello="world") ... ) == foo(1, 2, 3, bar=4, hello="world") >>> assert wfoo(1, (2, 3), bar=4) == foo(1, 2, 3, bar=4) >>> assert wfoo(1, (), bar=4) == foo(1, bar=4) ``` Note that if there is not variadic positional arguments, the variadic keyword will still be a keyword-only kind. ```pycon >>> @ch_variadics_to_non_variadic_kind ... def func(a, bar=None, **kwargs): ... return f"{a=}, {bar=}, {kwargs=}" >>> str(Sig(func)) '(a, bar=None, *, kwargs={})' >>> assert func(1, bar=4, kwargs=dict(hello="world") ... ) == "a=1, bar=4, kwargs={'hello': 'world'}" ``` If the function has neither variadic kinds, it will remain untouched. ```pycon >>> def func(a, /, b, *, c=3): ... return a + b + c >>> ch_variadics_to_non_variadic_kind(func) == func True ``` If you only want the variadic positional to be handled, but leave leave any VARIADIC_KEYWORD kinds (`**kwargs`) alone, you can do so by setting `ch_variadic_keyword_to_keyword=False`. If you’ll need to use `ch_variadics_to_non_variadic_kind` in such a way repeatedly, we suggest you use `functools.partial` to not have to specify this configuration repeatedly. ```pycon >>> from functools import partial >>> tuple_the_args = partial(ch_variadics_to_non_variadic_kind, ... ch_variadic_keyword_to_keyword=False ... ) >>> @tuple_the_args ... def foo(a, *args, bar=None, **kwargs): ... return f"{a=}, {args=}, {bar=}, {kwargs=}" >>> Sig(foo) >>> foo(1, (2, 3), bar=4, hello="world") "a=1, args=(2, 3), bar=4, kwargs={'hello': 'world'}" ``` ### dol.signatures.common_and_diff_argnames(func1, func2) Get list of argument names that are common to two functions, as well as the two lists of names that are different * **Parameters:** * **func1** (`callable`) – First function * **func2** (`callable`) – Second function * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** A dict with fields ‘common’, ‘func1_not_func2’, and ‘func2_not_func1’ ```pycon >>> def f(t, h, i, n, k): ... ... ... >>> def g(t, w, i, c, e): ... ... ... >>> common_and_diff_argnames(f, g) {'common': ['t', 'i'], 'func1_not_func2': ['h', 'n', 'k'], 'func2_not_func1': ['w', 'c', 'e']} >>> common_and_diff_argnames(g, f) {'common': ['t', 'i'], 'func1_not_func2': ['w', 'c', 'e'], 'func2_not_func1': ['h', 'n', 'k']} ``` ### dol.signatures.copy_func(f) Copy a function (not sure it works with all types of callables) ### dol.signatures.defaults_are_the_same_when_not_empty(dflt1, dflt2) Check if two defaults are the same when they are not empty. ```pycon >>> defaults_are_the_same_when_not_empty(1, 1) True >>> defaults_are_the_same_when_not_empty(1, 2) False >>> defaults_are_the_same_when_not_empty(1, None) False >>> defaults_are_the_same_when_not_empty(1, Parameter.empty) True ``` ### dol.signatures.dflt1_is_empty_or_dflt2_is_not(dflt1, dflt2) Why such a strange default comparison function? This is to be used as a default in is_call_compatible_with. Consider two functions func1 and func2 with a parameter p with default values dflt1 and dflt2 respectively. If dflt1 was not empty and dflt2 was, this would mean that func1 could be called without specifying p, but func2 couldn’t. So to avoid this situation, we use dflt1_is_empty_or_dflt2_is_not as the default ### dol.signatures.dflt1_is_empty_or_dflt2_is_not_param_comparator(param1, param2, \*, name=, kind=, default=, annotation=, aggreg=) Permissive version of param_comparator that ignores any differences of parameter attributes. It is meant to be used with partial, but with a permissive base, contrary to the base param_comparator which requires strict equality (`eq`) for all attributes. * **Return type:** *Comparison* ### dol.signatures.dict_of_attribute_signatures(cls) A function that extracts the signatures of all callable attributes of a class. * **Parameters:** **cls** ([`type`](https://docs.python.org/3/builtins/functions.html#type)) – The class that holds the the `(name, func)` pairs we want to extract. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature)] * **Returns:** A dict of `(name, signature(func))` pairs extracted from class. One of the intended applications is to use `dict_of_attribute_signatures` as a decorator, like so: ```pycon >>> @dict_of_attribute_signatures ... class names_and_signatures: ... def foo(x: str, *, y=2) -> tuple: ... ... def bar(z, /) -> float: ... >>> names_and_signatures {'foo': tuple>, 'bar': float>} ``` ### dol.signatures.ensure_params(obj=None) Get an interable of Parameter instances from an object. * **Parameters:** **obj** (`Union`[[`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) * **Returns:** From a callable: ```pycon >>> def f(w, /, x: float = 1, y=1, *, z: int = 1): ... ... >>> ensure_params(f) [, , , ] ``` From an iterable of strings, dicts, or tuples ```pycon >>> ensure_params( ... [ ... "xyz", ... ( ... "b", ... Parameter.empty, ... int, ... ), # if you want an annotation without a default use Parameter.empty ... ( ... "c", ... 2, ... ), # if you just want a default, make it the second element of your tup ... dict(name="d", kind=Parameter.VAR_KEYWORD), ... ] ... ) # all kinds are by default PK: Use dict to specify otherwise. [, , , ] ``` If no input is given, an empty list is returned. ```pycon >>> ensure_params() # equivalent to ensure_params(None) [] ``` ### dol.signatures.extract_arguments(params, , what_to_do_with_remainding='return', include_all_when_var_keywords_in_params=False, assert_no_missing_position_only_args=False, \*\*kwargs) Extract arguments needed to satisfy the params of a callable, dealing with the dirty details. Returns an (param_args, param_kwargs, remaining_kwargs) tuple where - param_args are the values of kwargs that are PO (POSITION_ONLY) as defined by params, - param_kwargs are those names that are both in params and not in param_args, and - remaining_kwargs are the remaining. Intended usage: When you need to call a function `func` that has some position-only arguments, but you have a kwargs dict of arguments in your hand. You can’t just to `func( **kwargs)`. But you can (now) do ```python # extract from kwargs what you need for func args, kwargs, remaining = extract_arguments(kwargs, func) # ... check if remaining is empty (or not, depending on your paranoia), # and then call the func: func(*args, **kwargs) ``` (And if you doing that a lot: Do put it in a decorator!) #### SEE ALSO extract_arguments.without_remainding The most frequent case you’ll encounter is when there’s no POSITION_ONLY args, your param_args will be empty and you param_kwargs will contain all the arguments that match params, in the order of these params. ```pycon >>> from inspect import signature >>> def f(a, b, c=None, d=0): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((), {'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'extra': 'stuff'}) ``` But sometimes you do have POSITION_ONLY arguments. What extract_arguments will do for you is return the value of these as the first element of the triple. ```pycon >>> def f(a, b, c=None, /, d=0): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4}, {'extra': 'stuff'}) ``` Note above how we get `(1, 2, 3)`, the order defined by the func’s signature, instead of `(2, 1, 3)`, the order defined by the kwargs. So it’s the params (e.g. function signature) that determine the order, not kwargs. When using to call a function, this is especially crucial if we use POSITION_ONLY arguments. See also that the third output, the remaining_kwargs, as `{'extra': 'stuff'}` since it was not in the params of the function. Even if you include a VAR_KEYWORD kind of argument in the function, it won’t change this behavior. ```pycon >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4}, {'extra': 'stuff'}) ``` This is because we don’t want to assume that all the kwargs can actually be included in a call to the function behind the params. Instead, the user can chose whether to include the remainder by doing a: ```python param_kwargs.update(remaining_kwargs) ``` et voilà. That said, we do understand that it may be a common pattern, so we’ll do that extra step for you if you specify `include_all_when_var_keywords_in_params=True`. ```pycon >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> extract_arguments( ... f, ... b=2, ... a=1, ... c=3, ... d=4, ... extra="stuff", ... include_all_when_var_keywords_in_params=True, ... ) ((1, 2, 3), {'d': 4, 'extra': 'stuff'}, {}) ``` If you’re expecting no remainder you might want to just get the args and kwargs ( not this third expected-to-be-empty remainder). You have two ways to do that, specifying: - `what_to_do_with_remainding='ignore'`, which will just return the (args, kwargs) pair - `what_to_do_with_remainding='assert_empty'`, which will do the same, but first assert the remainder is empty We suggest to use `functools.partial` to configure the `argument_argument` you need. ```pycon >>> from functools import partial >>> arg_extractor = partial( ... extract_arguments, ... what_to_do_with_remainding="assert_empty", ... include_all_when_var_keywords_in_params=True, ... ) >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> arg_extractor(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4, 'extra': 'stuff'}) ``` And what happens if the kwargs doesn’t contain all the POSITION_ONLY arguments? ```pycon >>> def f(a, b, c=None, /, d=0): ... ... ... >>> extract_arguments(f, b=2, d="is a kw arg", e="is not an arg at all") ((MissingArgValFor("a"), 2, MissingArgValFor("c")), {'d': 'is a kw arg'}, {'e': 'is not an arg at all'}) ``` A few more examples… Let’s call `extract_arguments` with params being not a function, but, a Signature instance, a mapping whose values are Parameter instances, or an iterable of Parameter instances… ```pycon >>> def func(a, b, /, c=None, *, d=0, **kws): ... ... ... >>> sig = Signature.from_callable(func) >>> param_map = sig.parameters >>> param_iterable = param_map.values() >>> kwargs = dict(b=2, a=1, c=3, d=4, extra="stuff") >>> assert extract_arguments(sig, **kwargs) == extract_arguments(func, **kwargs) >>> assert extract_arguments(param_map, **kwargs) == extract_arguments( ... func, **kwargs ... ) >>> assert extract_arguments(param_iterable, **kwargs) == extract_arguments( ... func, **kwargs ... ) ``` Edge case: No params specified? No problem. You’ll just get empty args and kwargs. Everything in the remainder ```pycon >>> extract_arguments(params=(), b=2, a=1, c=3, d=0) ((), {}, {'b': 2, 'a': 1, 'c': 3, 'd': 0}) ``` * **Parameters:** * **params** (`Union`[[`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Specifies what PO arguments should be extracted. Could be a callable, Signature, iterable of Parameters… * **what_to_do_with_remainding** – ‘return’ (default): function will return `param_args`, `param_kwargs`, `remaining_kwargs` ‘ignore’: function will return `param_args`, `param_kwargs` ‘assert_empty’: function will assert that `remaining_kwargs` is empty and then return `param_args`, `param_kwargs` * **include_all_when_var_keywords_in_params** – If True and `params` has a VAR_KEYWORD parameter, the remaining kwargs are merged into `param_kwargs` (leaving an empty remainder). * **assert_no_missing_position_only_args** – If True, assert that no position-only argument is missing from `kwargs`. * **kwargs** – The kwargs to extract the args from * **Returns:** A (param_args, param_kwargs, remaining_kwargs) tuple. ### dol.signatures.extract_arguments_asserting_no_remainder(params, , what_to_do_with_remainding='assert_empty', include_all_when_var_keywords_in_params=False, assert_no_missing_position_only_args=False, \*\*kwargs) Extract arguments needed to satisfy the params of a callable, dealing with the dirty details. Returns an (param_args, param_kwargs, remaining_kwargs) tuple where - param_args are the values of kwargs that are PO (POSITION_ONLY) as defined by params, - param_kwargs are those names that are both in params and not in param_args, and - remaining_kwargs are the remaining. Intended usage: When you need to call a function `func` that has some position-only arguments, but you have a kwargs dict of arguments in your hand. You can’t just to `func( **kwargs)`. But you can (now) do ```python # extract from kwargs what you need for func args, kwargs, remaining = extract_arguments(kwargs, func) # ... check if remaining is empty (or not, depending on your paranoia), # and then call the func: func(*args, **kwargs) ``` (And if you doing that a lot: Do put it in a decorator!) #### SEE ALSO extract_arguments.without_remainding The most frequent case you’ll encounter is when there’s no POSITION_ONLY args, your param_args will be empty and you param_kwargs will contain all the arguments that match params, in the order of these params. ```pycon >>> from inspect import signature >>> def f(a, b, c=None, d=0): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((), {'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'extra': 'stuff'}) ``` But sometimes you do have POSITION_ONLY arguments. What extract_arguments will do for you is return the value of these as the first element of the triple. ```pycon >>> def f(a, b, c=None, /, d=0): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4}, {'extra': 'stuff'}) ``` Note above how we get `(1, 2, 3)`, the order defined by the func’s signature, instead of `(2, 1, 3)`, the order defined by the kwargs. So it’s the params (e.g. function signature) that determine the order, not kwargs. When using to call a function, this is especially crucial if we use POSITION_ONLY arguments. See also that the third output, the remaining_kwargs, as `{'extra': 'stuff'}` since it was not in the params of the function. Even if you include a VAR_KEYWORD kind of argument in the function, it won’t change this behavior. ```pycon >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4}, {'extra': 'stuff'}) ``` This is because we don’t want to assume that all the kwargs can actually be included in a call to the function behind the params. Instead, the user can chose whether to include the remainder by doing a: ```python param_kwargs.update(remaining_kwargs) ``` et voilà. That said, we do understand that it may be a common pattern, so we’ll do that extra step for you if you specify `include_all_when_var_keywords_in_params=True`. ```pycon >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> extract_arguments( ... f, ... b=2, ... a=1, ... c=3, ... d=4, ... extra="stuff", ... include_all_when_var_keywords_in_params=True, ... ) ((1, 2, 3), {'d': 4, 'extra': 'stuff'}, {}) ``` If you’re expecting no remainder you might want to just get the args and kwargs ( not this third expected-to-be-empty remainder). You have two ways to do that, specifying: - `what_to_do_with_remainding='ignore'`, which will just return the (args, kwargs) pair - `what_to_do_with_remainding='assert_empty'`, which will do the same, but first assert the remainder is empty We suggest to use `functools.partial` to configure the `argument_argument` you need. ```pycon >>> from functools import partial >>> arg_extractor = partial( ... extract_arguments, ... what_to_do_with_remainding="assert_empty", ... include_all_when_var_keywords_in_params=True, ... ) >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> arg_extractor(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4, 'extra': 'stuff'}) ``` And what happens if the kwargs doesn’t contain all the POSITION_ONLY arguments? ```pycon >>> def f(a, b, c=None, /, d=0): ... ... ... >>> extract_arguments(f, b=2, d="is a kw arg", e="is not an arg at all") ((MissingArgValFor("a"), 2, MissingArgValFor("c")), {'d': 'is a kw arg'}, {'e': 'is not an arg at all'}) ``` A few more examples… Let’s call `extract_arguments` with params being not a function, but, a Signature instance, a mapping whose values are Parameter instances, or an iterable of Parameter instances… ```pycon >>> def func(a, b, /, c=None, *, d=0, **kws): ... ... ... >>> sig = Signature.from_callable(func) >>> param_map = sig.parameters >>> param_iterable = param_map.values() >>> kwargs = dict(b=2, a=1, c=3, d=4, extra="stuff") >>> assert extract_arguments(sig, **kwargs) == extract_arguments(func, **kwargs) >>> assert extract_arguments(param_map, **kwargs) == extract_arguments( ... func, **kwargs ... ) >>> assert extract_arguments(param_iterable, **kwargs) == extract_arguments( ... func, **kwargs ... ) ``` Edge case: No params specified? No problem. You’ll just get empty args and kwargs. Everything in the remainder ```pycon >>> extract_arguments(params=(), b=2, a=1, c=3, d=0) ((), {}, {'b': 2, 'a': 1, 'c': 3, 'd': 0}) ``` * **Parameters:** * **params** – Specifies what PO arguments should be extracted. Could be a callable, Signature, iterable of Parameters… * **what_to_do_with_remainding** – ‘return’ (default): function will return `param_args`, `param_kwargs`, `remaining_kwargs` ‘ignore’: function will return `param_args`, `param_kwargs` ‘assert_empty’: function will assert that `remaining_kwargs` is empty and then return `param_args`, `param_kwargs` * **include_all_when_var_keywords_in_params** – If True and `params` has a VAR_KEYWORD parameter, the remaining kwargs are merged into `param_kwargs` (leaving an empty remainder). * **assert_no_missing_position_only_args** – If True, assert that no position-only argument is missing from `kwargs`. * **kwargs** – The kwargs to extract the args from * **Returns:** A (param_args, param_kwargs, remaining_kwargs) tuple. ### dol.signatures.extract_arguments_ignoring_remainder(params, , what_to_do_with_remainding='ignore', include_all_when_var_keywords_in_params=False, assert_no_missing_position_only_args=False, \*\*kwargs) Extract arguments needed to satisfy the params of a callable, dealing with the dirty details. Returns an (param_args, param_kwargs, remaining_kwargs) tuple where - param_args are the values of kwargs that are PO (POSITION_ONLY) as defined by params, - param_kwargs are those names that are both in params and not in param_args, and - remaining_kwargs are the remaining. Intended usage: When you need to call a function `func` that has some position-only arguments, but you have a kwargs dict of arguments in your hand. You can’t just to `func( **kwargs)`. But you can (now) do ```python # extract from kwargs what you need for func args, kwargs, remaining = extract_arguments(kwargs, func) # ... check if remaining is empty (or not, depending on your paranoia), # and then call the func: func(*args, **kwargs) ``` (And if you doing that a lot: Do put it in a decorator!) #### SEE ALSO extract_arguments.without_remainding The most frequent case you’ll encounter is when there’s no POSITION_ONLY args, your param_args will be empty and you param_kwargs will contain all the arguments that match params, in the order of these params. ```pycon >>> from inspect import signature >>> def f(a, b, c=None, d=0): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((), {'a': 1, 'b': 2, 'c': 3, 'd': 4}, {'extra': 'stuff'}) ``` But sometimes you do have POSITION_ONLY arguments. What extract_arguments will do for you is return the value of these as the first element of the triple. ```pycon >>> def f(a, b, c=None, /, d=0): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4}, {'extra': 'stuff'}) ``` Note above how we get `(1, 2, 3)`, the order defined by the func’s signature, instead of `(2, 1, 3)`, the order defined by the kwargs. So it’s the params (e.g. function signature) that determine the order, not kwargs. When using to call a function, this is especially crucial if we use POSITION_ONLY arguments. See also that the third output, the remaining_kwargs, as `{'extra': 'stuff'}` since it was not in the params of the function. Even if you include a VAR_KEYWORD kind of argument in the function, it won’t change this behavior. ```pycon >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> extract_arguments(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4}, {'extra': 'stuff'}) ``` This is because we don’t want to assume that all the kwargs can actually be included in a call to the function behind the params. Instead, the user can chose whether to include the remainder by doing a: ```python param_kwargs.update(remaining_kwargs) ``` et voilà. That said, we do understand that it may be a common pattern, so we’ll do that extra step for you if you specify `include_all_when_var_keywords_in_params=True`. ```pycon >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> extract_arguments( ... f, ... b=2, ... a=1, ... c=3, ... d=4, ... extra="stuff", ... include_all_when_var_keywords_in_params=True, ... ) ((1, 2, 3), {'d': 4, 'extra': 'stuff'}, {}) ``` If you’re expecting no remainder you might want to just get the args and kwargs ( not this third expected-to-be-empty remainder). You have two ways to do that, specifying: - `what_to_do_with_remainding='ignore'`, which will just return the (args, kwargs) pair - `what_to_do_with_remainding='assert_empty'`, which will do the same, but first assert the remainder is empty We suggest to use `functools.partial` to configure the `argument_argument` you need. ```pycon >>> from functools import partial >>> arg_extractor = partial( ... extract_arguments, ... what_to_do_with_remainding="assert_empty", ... include_all_when_var_keywords_in_params=True, ... ) >>> def f(a, b, c=None, /, d=0, **kws): ... ... ... >>> arg_extractor(f, b=2, a=1, c=3, d=4, extra="stuff") ((1, 2, 3), {'d': 4, 'extra': 'stuff'}) ``` And what happens if the kwargs doesn’t contain all the POSITION_ONLY arguments? ```pycon >>> def f(a, b, c=None, /, d=0): ... ... ... >>> extract_arguments(f, b=2, d="is a kw arg", e="is not an arg at all") ((MissingArgValFor("a"), 2, MissingArgValFor("c")), {'d': 'is a kw arg'}, {'e': 'is not an arg at all'}) ``` A few more examples… Let’s call `extract_arguments` with params being not a function, but, a Signature instance, a mapping whose values are Parameter instances, or an iterable of Parameter instances… ```pycon >>> def func(a, b, /, c=None, *, d=0, **kws): ... ... ... >>> sig = Signature.from_callable(func) >>> param_map = sig.parameters >>> param_iterable = param_map.values() >>> kwargs = dict(b=2, a=1, c=3, d=4, extra="stuff") >>> assert extract_arguments(sig, **kwargs) == extract_arguments(func, **kwargs) >>> assert extract_arguments(param_map, **kwargs) == extract_arguments( ... func, **kwargs ... ) >>> assert extract_arguments(param_iterable, **kwargs) == extract_arguments( ... func, **kwargs ... ) ``` Edge case: No params specified? No problem. You’ll just get empty args and kwargs. Everything in the remainder ```pycon >>> extract_arguments(params=(), b=2, a=1, c=3, d=0) ((), {}, {'b': 2, 'a': 1, 'c': 3, 'd': 0}) ``` * **Parameters:** * **params** – Specifies what PO arguments should be extracted. Could be a callable, Signature, iterable of Parameters… * **what_to_do_with_remainding** – ‘return’ (default): function will return `param_args`, `param_kwargs`, `remaining_kwargs` ‘ignore’: function will return `param_args`, `param_kwargs` ‘assert_empty’: function will assert that `remaining_kwargs` is empty and then return `param_args`, `param_kwargs` * **include_all_when_var_keywords_in_params** – If True and `params` has a VAR_KEYWORD parameter, the remaining kwargs are merged into `param_kwargs` (leaving an empty remainder). * **assert_no_missing_position_only_args** – If True, assert that no position-only argument is missing from `kwargs`. * **kwargs** – The kwargs to extract the args from * **Returns:** A (param_args, param_kwargs, remaining_kwargs) tuple. ### dol.signatures.has_signature(obj, robust=False) Check if an object has a signature – i.e. is callable and inspect.signature( obj) returns something. This can be used to more easily get signatures in bulk without having to write try/catches: ```pycon >>> from functools import partial >>> len( ... list( ... filter( ... None, ... map( ... partial(has_signature, robust=False), ... (Sig, print, map, filter, Sig.wrap), ... ), ... ) ... ) ... ) 2 ``` If robust is set to True, `has_signature` will use `Sig` to get the signature, so will return True in most cases. ### dol.signatures.insert_annotations(s, , , return_annotation, \*\*annotations) Insert annotations in a signature. (Note: not really insert but returns a copy of input signature) ```pycon >>> from inspect import signature >>> s = signature(lambda a, b, c=1, d="bar": 0) >>> s >>> ss = insert_annotations(s, b=int, d=str) >>> ss >>> insert_annotations(s, b=int, d=str, e=list) Traceback (most recent call last): ... AssertionError: These argument names weren't found in the signature: {'e'} ``` ### dol.signatures.is_call_compatible_with(sig1, sig2, , param_comparator=None) Return True if `sig1` is compatible with `sig2`. Meaning that all valid ways to call `sig1` are valid for `sig2`. * **Parameters:** * **sig1** ([`Sig`](_autosummary/dol.signatures.html.md#dol.signatures.Sig)) – The main signature. * **sig2** ([`Sig`](_autosummary/dol.signatures.html.md#dol.signatures.Sig)) – The signature to be compared with. * **param_comparator** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter), [`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The function used to compare two parameters * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ```pycon >>> is_call_compatible_with( ... Sig('(a, /, b, *, c)'), ... Sig('(a, b, c)') ... ) True >>> is_call_compatible_with( ... Sig('()'), ... Sig('(a)') ... ) False >>> is_call_compatible_with( ... Sig('()'), ... Sig('(a=0)') ... ) True >>> is_call_compatible_with( ... Sig('(a, /, *, c)'), ... Sig('(a, /, b, *, c)') ... ) False >>> is_call_compatible_with( ... Sig('(a, /, *, c)'), ... Sig('(a, /, b=0, *, c)') ... ) True >>> is_call_compatible_with( ... Sig('(a, /, b)'), ... Sig('(a, /, b, *, c)') ... ) False >>> is_call_compatible_with( ... Sig('(a, /, b)'), ... Sig('(a, /, b, *, c=0)') ... ) True >>> is_call_compatible_with( ... Sig('(a, /, b, *, c)'), ... Sig('(*args, **kwargs)') ... ) True ``` ### dol.signatures.is_signature_error(e) Check if an exception is a signature error * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### dol.signatures.keyed_comparator(comparator, key) Create a key-function enabled binary operator. In various places in python functionality is extended by allowing a key function. For example, the `sorted` function allows a key function to be passed, which is applied to each element before sorting. The keyed_comparator function allows a comparator to be extended in the same way. The returned comparator will apply the key function toeach input before applying the original comparator. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)] ```pycon >>> from operator import eq >>> parity = lambda x: x % 2 >>> comparator = keyed_comparator(eq, parity) >>> list(map(comparator, [1, 1, 2, 2], [3, 4, 5, 6])) [True, False, False, True] ``` ### dol.signatures.kind_forgiving_func(func, kinds_modifier=) Wraps the func, changing the argument kinds according to kinds_modifier. The default behaviour is to change all kinds to POSITIONAL_OR_KEYWORD kinds. The original purpose of this function is to remove argument-kind restriction annoyances when doing functional manipulations such as: ```pycon >>> from functools import partial >>> isinstance_of_str = partial(isinstance, class_or_tuple=str) >>> isinstance_of_str('I am a string') Traceback (most recent call last): ... TypeError: isinstance() takes no keyword arguments ``` Here, instead, we can just get a kinder version of the function and do what we want to do: ```pycon >>> _isinstance = kind_forgiving_func(isinstance) >>> isinstance_of_str = partial(_isinstance, class_or_tuple=str) >>> isinstance_of_str('I am a string') True >>> isinstance_of_str(42) False ``` #### SEE ALSO `i2.signatures.all_pk_signature` ### dol.signatures.mk_sig_from_args(\*args_without_default, \*\*args_with_defaults) Make a Signature instance by specifying args_without_default and args_with_defaults. ```pycon >>> mk_sig_from_args("a", "b", c=1, d="bar") ``` ### dol.signatures.name_of_obj(o, \*, base_name_of_obj=operator.attrgetter('_\_name_\_'), caught_exceptions=(, ), default_factory=) Tries to find the (or “a”) name for an object, even if `__name__` doesn’t exist. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ```pycon >>> name_of_obj(map) 'map' >>> name_of_obj([1, 2, 3]) 'list' >>> name_of_obj(print) 'print' >>> name_of_obj(lambda x: x) '' >>> from functools import partial >>> name_of_obj(partial(print, sep=",")) 'print' >>> from functools import cached_property >>> class A: ... @property ... def prop(self): ... return 1.0 ... @cached_property ... def cached_prop(self): ... return 2.0 >>> name_of_obj(A.prop) 'prop' >>> name_of_obj(A.cached_prop) 'cached_prop' ``` Note that `name_of_obj` uses the `__name__` attribute as its base way to get a name. You can customize this behavior though. For example, see that: ```pycon >>> from inspect import Signature >>> name_of_obj(Signature.replace) 'replace' ``` If you want to get the fully qualified name of an object, you can do: ```pycon >>> alt = partial(name_of_obj, base_name_of_obj=attrgetter('__qualname__')) >>> alt(Signature.replace) 'Signature.replace' ``` ### dol.signatures.param_binary_func(param1, param2, \*, name=, kind=, default=, annotation=, aggreg=) Compare two parameters. Note that by default, this function is strict, and will return False if any of the parameters are not equal. This is because the default aggregation function is `all` and the default comparison functions of the parameter’s attributes are `eq` (meaning equality, not identity). But you can change that by passing different comparison functions and/or aggregation functions. In fact, the real purpose of this function is to be used as a factory of parameter binary functions, through parametrizing it with `functools.partial`. The parameter binary functions themselves are meant to be used to make signature binary functions. * **Parameters:** * **param1** ([`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)) – first parameter * **param2** ([`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)) – second parameter * **name** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare names * **kind** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare kinds * **default** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare defaults * **annotation** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare annotations * **aggreg** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – function to aggregate results * **Return type:** [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`) ```pycon >>> from inspect import Parameter >>> param1 = Parameter('x', Parameter.POSITIONAL_OR_KEYWORD) >>> param2 = Parameter('x', Parameter.POSITIONAL_OR_KEYWORD) >>> param_binary_func(param1, param2) True ``` See [https://github.com/i2mint/i2/issues/50#issuecomment-1381686812](https://github.com/i2mint/i2/issues/50#issuecomment-1381686812) for discussion. ### dol.signatures.param_comparator(param1, param2, \*, name=, kind=, default=, annotation=, aggreg=) Compare two parameters. Note that by default, this function is strict, and will return False if any of the parameters are not equal. This is because the default aggregation function is `all` and the default comparison functions of the parameter’s attributes are `eq` (meaning equality, not identity). But you can change that by passing different comparison functions and/or aggregation functions. In fact, the real purpose of this function is to be used as a factory of parameter binary functions, through parametrizing it with `functools.partial`. The parameter binary functions themselves are meant to be used to make signature binary functions. * **Parameters:** * **param1** ([`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)) – first parameter * **param2** ([`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)) – second parameter * **name** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare names * **kind** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare kinds * **default** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare defaults * **annotation** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Compared`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]) – function to compare annotations * **aggreg** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`)]], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – function to aggregate results * **Return type:** [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Comparison`) ```pycon >>> from inspect import Parameter >>> param1 = Parameter('x', Parameter.POSITIONAL_OR_KEYWORD) >>> param2 = Parameter('x', Parameter.POSITIONAL_OR_KEYWORD) >>> param_binary_func(param1, param2) True ``` See [https://github.com/i2mint/i2/issues/50#issuecomment-1381686812](https://github.com/i2mint/i2/issues/50#issuecomment-1381686812) for discussion. ### dol.signatures.param_comparison_dict(param1, param2, \*, name=, kind=, default=, annotation=, aggreg=) A ParamComparator that returns a dictionary with pairs parameter attributes. ```pycon >>> param1 = Sig('(a: int = 1)')['a'] >>> param2 = Sig('(a: str = 2)')['a'] >>> param_comparison_dict(param1, param2) {'name': ('a', 'a'), 'kind': ..., 'default': (1, 2), 'annotation': (, )} ``` * **Return type:** *Comparison* ### dol.signatures.param_differences_dict(param1, param2, \*, name=, kind=, default=, annotation=) Makes a dictionary exibiting the differences between two parameters. ```pycon >>> param1 = Sig('(a: int = 1)')['a'] >>> param2 = Sig('(a: str = 2)')['a'] >>> param_differences_dict(param1, param2) {'default': (1, 2), 'annotation': (, )} >>> param_differences_dict(param1, param2, default=lambda x, y: isinstance(x, type(y))) {'annotation': (, )} ``` ### dol.signatures.param_for_kind(name=None, kind='positional_or_keyword', with_default=False, annotation) Function to easily and flexibly make inspect.Parameter objects for testing. It’s annoying to have to compose parameters from scratch to testing things. This tool should help making it less annoying. ```pycon >>> list(map(param_for_kind, param_kinds)) [, , , , ] >>> param_for_kind.positional_or_keyword() >>> param_for_kind.positional_or_keyword("foo") >>> param_for_kind.keyword_only() >>> param_for_kind.keyword_only("baz", with_default=True) ``` ### dol.signatures.permissive_param_comparator(param1, param2, \*, name=, kind=, default=, annotation=, aggreg=) Permissive version of param_comparator that ignores any differences of parameter attributes. It is meant to be used with partial, but with a permissive base, contrary to the base param_comparator which requires strict equality (`eq`) for all attributes. * **Return type:** *Comparison* ### dol.signatures.postprocess(egress) A decorator that will process the output of the wrapped function with egress ### dol.signatures.replace_kwargs_using(sig) Decorator that replaces the variadic keyword argument of the target function using the `sig`, the signature of a source function. It essentially injects the difference between `sig` and the target function’s signature into the target function’s signature. That is, it replaces the variadic keyword argument (a.k.a. “kwargs”) with those parameters that are in `sig` but not in the target function’s signature. This is meant to be used when a `targ_func` (the function you’ll apply the decorator to) has a variadict keyword argument that is just used to forward “extra” arguments to another function, and you want to make sure that the signature of the `targ_func` is consistent with the `sig` signature. (Also, you don’t want to copy the signatures around manually.) In the following, `sauce` (the target function) has a variadic keyword argument, `sauce_kwargs`, that is used to forward extra arguments to `apple` (the source function). ```pycon >>> def apple(a, x: int, y=2, *, z=3, **extra_apple_options): ... return a + x + y + z >>> @replace_kwargs_using(apple) ... def sauce(a, b, c, **sauce_kwargs): ... return b * c + apple(a, **sauce_kwargs) ``` The function will works: ```pycon >>> sauce(1, 2, 3, x=4, z=5) # func still works? Should be: 1 + 4 + 2 + 5 + 2 * 3 18 ``` But the signature now doesn’t have the `**sauce_kwargs`, but more informative signature elements sourced from `apple`: ```pycon >>> Sig(sauce) ``` One thing to note is that the order of the arguments in the signature of `apple` may change to accomodate for the python parameter order rules (see [https://docs.python.org/3/reference/compound_stmts.html#function-definitions](https://docs.python.org/3/reference/compound_stmts.html#function-definitions)). The new order will try to conserve the order of the original arguments of `sauce` in-so-far as it doesn’t violate the python parameter order rules, though. See examples below: ```pycon >>> @Sig.replace_kwargs_using(apple) ... def sauce(a, b=2, c=3, **sauce_kwargs): ... return b * c + apple(a, **sauce_kwargs) >>> Sig(sauce) ``` ```pycon >>> @Sig.replace_kwargs_using(apple) ... def sauce(a=1, b=2, c=3, **sauce_kwargs): ... return b * c + apple(a, **sauce_kwargs) >>> Sig(sauce) ``` ### dol.signatures.resolve_function(obj) Get the underlying function of a property or cached_property Note that if all conditions fail, the object itself is returned. The problem this function solves is that sometimes there’s a function behind an object, but it’s not always easy to get to it. For example, in a class, you might want to get the source of the code decorated with `@property`, a `@cached_property`, or a `partial` function. Consider the following example: * **Return type:** `Union`[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`T`), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)] ```pycon >>> from functools import cached_property, partial >>> class C: ... @property ... def prop(self): ... pass ... @cached_property ... def cached_prop(self): ... pass ... partial_func = partial(partial) ``` Note that `prop` is not callable, and you can’t get its source. ```pycon >>> import inspect >>> callable(C.prop) False >>> inspect.getsource(C.prop) Traceback (most recent call last): ... TypeError: is not a module, class, method, function, traceback, frame, or code object ``` But if you grab the underlying function, you can get the source: ```pycon >>> func = resolve_function(C.prop) >>> callable(func) True >>> isinstance(inspect.getsource(func), str) True ``` Same goes with `cached_property` and `partial`: ```pycon >>> isinstance(inspect.getsource(resolve_function(C.cached_prop)), str) True >>> isinstance(inspect.getsource(resolve_function(C.partial_func)), str) True ``` ### dol.signatures.set_signature_of_func(func, parameters, , return_annotation, \_\_validate_parameters_\_=True) Set the signature of a function, with sugar. * **Parameters:** * **func** – Function whose signature you want to set * **signature** – A list of parameter specifications. This could be an * **that** (*inspect.Parameter object* *or* *anything*) – the mk_param function can resolve into an inspect.Parameter object. * **return_annotation** – Passed on to inspect.Signature. * **\_\_validate_parameters_\_** – Passed on to inspect.Signature. * **Returns:** None (but sets the signature of the input function) ```pycon >>> import inspect >>> def foo(*args, **kwargs): ... pass ... >>> inspect.signature(foo) >>> set_signature_of_func(foo, ["a", "b", "c"]) >>> inspect.signature(foo) >>> set_signature_of_func( ... foo, ["a", ("b", None), ("c", 42, int)] ... ) # specifying defaults and annotations >>> inspect.signature(foo) >>> set_signature_of_func( ... foo, ["a", "b", "c"], return_annotation=str ... ) # specifying return annotation >>> inspect.signature(foo) str> >>> # But you can always specify parameters the "long" way >>> set_signature_of_func( ... foo, ... [inspect.Parameter(name="kws", kind=inspect.Parameter.VAR_KEYWORD)], ... return_annotation=str, ... ) >>> inspect.signature(foo) str> ``` ### dol.signatures.sig_to_dataclass(sig, , cls_name=None, bases=(), module=None, \*\*kwargs) Make a `class` (through `make_dataclass`) from the given signature. * **Parameters:** * **sig** (`Union`[[`Signature`](https://docs.python.org/3/library/inspect.html#inspect.Signature), [`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Parameter`](https://docs.python.org/3/library/inspect.html#inspect.Parameter)], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A `SignatureAble`, that is, anything that ensure_signature can resolve into an `inspect.Signature` object, including a signature object itself, but also most callables, a list or params, etc. * **cls_name** – The same as `cls_name` of `dataclasses.make_dataclass` * **bases** – The same as `bases` of `dataclasses.make_dataclass` * **module** – Set to module (usually `__name__` to specify ther module of caller) so that the class and instances can be pickle-able. * **kwargs** – Passed on to `dataclasses.make_dataclass` * **Returns:** A dataclass ```pycon >>> def foo(a, /, b : int=2, *, c=3): ... pass ... >>> K = sig_to_dataclass(foo, cls_name='K') >>> str(Sig(K)) '(a, b: int = 2, c=3) -> None' >>> k = K(1,2,3) >>> (k.a, k.b, k.c) (1, 2, 3) ``` Would also work with any of these (and more): ```pycon >>> K = sig_to_dataclass(Sig(foo), cls_name='K') >>> K = sig_to_dataclass(Sig(foo).params, cls_name='K') ``` #### NOTE `cls_name` is not required (we’ll try to figure out a good default for you), but it’s advised to only use this convenience in extreme mode. Choosing your own name might make for a safer future if you’re reusing your class. ### dol.signatures.sort_params(params) * **Parameters:** **params** – An iterable of `Parameter` instances * **Returns:** A list of these instances sorted so as to obey the `kind` and `default` order rules of python signatures. Note 1: It doesn’t mean that these params constitute a valid signature together, since it doesn’t verify rules like unicity of names and variadic kinds. Note 2: Though you can use `sorted` on an iterable of `i2.signatures.Param` instances, know that even for sorting the three parameters below, the `sort_params` function is more than twice as fast. ```pycon >>> from inspect import Parameter >>> sort_params( ... [Parameter('a', kind=Parameter.POSITIONAL_OR_KEYWORD, default=1), ... Parameter('b', kind=Parameter.POSITIONAL_ONLY), ... Parameter('c', kind=Parameter.POSITIONAL_OR_KEYWORD)] ... ) [, , ] ``` ### dol.signatures.tuple_the_args(func, , ch_variadic_keyword_to_keyword=False) A decorator that will change a VAR_POSITIONAL (\*args) argument to a tuple (args) argument of the same name. ### dol.signatures.use_interface(interface_sig) Use interface_sig as (enforced/validated) signature of the decorated function. That is, the decorated function will use the original function has the backend, the function actually doing the work, but with a frontend specified (in looks and in argument validation) `interface_sig` consider the situation where are functionality is parametrized by a function `g` taking two inputs, `a`, and `b`. Now you want to carry out this functionality using a function `f` that does what `g` should do, but doesn’t use `a`, and doesn’t even have it in it’s arguments. The solution to this is to \_adapt_ `f` to the `g` interface: ```python def my_g(a, b): return f(a) ``` and use `my_g`. ```pycon >>> f = lambda a: a * 11 >>> interface = lambda a, b=None: ... >>> >>> new_f = use_interface(interface)(f) ``` See how only the first argument, or `a` keyword argument, is taken into account in `new_f`: ```pycon >>> assert new_f(2) == f(2) >>> assert new_f(2, 3) == f(2) >>> assert new_f(2, b=3) == f(2) >>> assert new_f(b=3, a=2) == f(2) ``` But if we add more positional arguments than `interface` allows, or any keyword arguments that `interface` doesn’t recognize… ```pycon >>> new_f(1,2,3) Traceback (most recent call last): ... TypeError: too many positional arguments >>> new_f(1, c=2) Traceback (most recent call last): ... TypeError: got an unexpected keyword argument 'c' ``` ### dol.signatures.validate_signature(func) Validates the signature of a function. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) ```pycon >>> @validate_signature ... def has_valid_signature(x=Sig.empty, y=2): ... pass >>> # all good, no errors raised >>> >>> @validate_signature ... def does_no_have_valid_signature(x=2, y=Sig.empty): ... pass Traceback (most recent call last): ... i2.signatures.InvalidSignature: Invalid signature for function : non-default argument follows default a rgument ``` # _autosummary/dol.sources.html.md # dol.sources Key-value views of disparate sources. Readers and persisters over things that are not stores to begin with: several stores at once (fan-out and cascades), sequences, functions, and the attributes of objects. Main entry points: - `FanoutReader`, `FanoutPersister`: one key, read from (written to) several stores - `CascadedStores`: write to all stores, read from the first one that has the key - `SequenceKvReader`: an iterable of elements, keyed by a key function (index by default) - `FuncReader`: functions as a store, keyed by name - `Attrs`: the attributes of an object as a (recursive) reader ```pycon >>> from dol.sources import FuncReader >>> def foo(): ... return 'bar' >>> r = FuncReader([foo]) >>> list(r), r['foo'] (['foo'], 'bar') ``` ### Functions | `ddir`(o) | | |---------------------------------|----| | `exclusive_subdict`(d, exclude) | | | `identity_func`(x) | | | `inclusive_subdict`(d, include) | | | `not_underscore_prefixed`(x) | | | `unique_element`(iterator) | | ### Classes | [`AttrContainer`](_autosummary/dol.sources.html.md#dol.sources.AttrContainer)(\*objects[, \_object_namer]) | Convenience class to hold Key-Val pairs as attribute-val pairs, with all the magic methods of mappings. | |----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`AttrDict`](_autosummary/dol.sources.html.md#dol.sources.AttrDict)(\*objects[, \_object_namer]) | Convenience class to hold Key-Val pairs with both a dict-like and struct-like interface. | | [`Attrs`](_autosummary/dol.sources.html.md#dol.sources.Attrs)(obj[, key_filt, getattrs]) | A simple recursive KvReader for the attributes of a python object. | | [`CachedKeysSequenceKvReader`](_autosummary/dol.sources.html.md#dol.sources.CachedKeysSequenceKvReader)(sequence[, key, ...]) | SequenceKvReader but with keys cached. | | [`CachedSequenceKvReader`](_autosummary/dol.sources.html.md#dol.sources.CachedSequenceKvReader)(sequence[, key, val, ...]) | SequenceKvReader but with the whole mapping cached as a dict. | | [`CascadedStores`](_autosummary/dol.sources.html.md#dol.sources.CascadedStores)(stores[, default, ...]) | A MutableMapping interface to a collection of stores that will write a value in all the stores it contains, read it from the first store it finds that has it, and write it back to all the stores up to the store where it found it. | | [`Ddir`](_autosummary/dol.sources.html.md#dol.sources.Ddir) | | | [`FanoutPersister`](_autosummary/dol.sources.html.md#dol.sources.FanoutPersister)(stores[, default, ...]) | A fanout persister is a fanout reader that can also set and delete items. | | [`FanoutReader`](_autosummary/dol.sources.html.md#dol.sources.FanoutReader)(stores[, default, ...]) | Get a 'fanout view' of a store of stores. | | [`FlatReader`](_autosummary/dol.sources.html.md#dol.sources.FlatReader)(readers) | Get a 'flat view' of a store of stores. | | [`FuncDag`](_autosummary/dol.sources.html.md#dol.sources.FuncDag)(funcs, \*\*kwargs) | | | [`FuncReader`](_autosummary/dol.sources.html.md#dol.sources.FuncReader)(funcs) | Reader that seeds itself from a data fetching function list Uses the function list names as the keys, and their returned value as the values. | | [`MultiSource`](_autosummary/dol.sources.html.md#dol.sources.MultiSource)(\*sources) | A read-only Mapping that composes multiple sources, tried in order. | | `ObjLoader`(data_of_key[, obj_of_data]) | | | [`ObjReader`](_autosummary/dol.sources.html.md#dol.sources.ObjReader)(_obj_of_key) | A reader that uses a specified function to get the contents for a given key. | | [`SequenceKvReader`](_autosummary/dol.sources.html.md#dol.sources.SequenceKvReader)(sequence[, key, val, ...]) | A KvReader that sources itself in an iterable of elements from which keys and values will be extracted and grouped by key. | ### Exceptions | [`NotUnique`](_autosummary/dol.sources.html.md#dol.sources.NotUnique) | Raised when an iterator was expected to have only one element, but had more | |--------------------------------------------------------------|-------------------------------------------------------------------------------| ### *class* dol.sources.AttrContainer(\*objects, \_object_namer=, \*\*named_objects) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Convenience class to hold Key-Val pairs as attribute-val pairs, with all the magic methods of mappings. On the other hand, you will not get the usuall non-dunders (non magic methods) of `Mappings`. This is so that you can use tab completion to access only the keys the container has, and not any of the non-dunder methods like `get`, `items`, etc. ```pycon >>> da = AttrContainer(foo='bar', life=42) >>> da.foo 'bar' >>> da['life'] 42 >>> da.true = 'love' >>> len(da) # count the number of fields 3 >>> da['friends'] = 'forever' # write as dict >>> da.friends # read as attribute 'forever' >>> list(da) # list fields (i.e. keys i.e. attributes) ['foo', 'life', 'true', 'friends'] >>> 'life' in da # check containement True ``` ```pycon >>> del da['friends'] # delete as dict >>> del da.foo # delete as attribute >>> list(da) ['life', 'true'] >>> da._source # the hidden Mapping (here dict) that is wrapped {'life': 42, 'true': 'love'} ``` If you don’t specify a name for some objects, `AttrContainer` will use the `__name__` attribute of the objects: ```pycon >>> d = AttrContainer(map, tuple, obj='objects') >>> list(d) ['map', 'tuple', 'obj'] ``` You can also specify a different way of auto naming the objects: ```pycon >>> d = AttrContainer('an', 'example', _object_namer=lambda x: f"_{len(x)}") >>> {k: getattr(d, k) for k in d} {'_2': 'an', '_7': 'example'} ``` #### SEE ALSO Objects in `py2store.utils.attr_dict` module ### *class* dol.sources.AttrDict(\*objects, \_object_namer=, \*\*named_objects) Bases: [`AttrContainer`](_autosummary/dol.sources.html.md#dol.sources.AttrContainer), [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister) Convenience class to hold Key-Val pairs with both a dict-like and struct-like interface. The dict-like interface has just the basic get/set/del/iter/len (all “dunders”: none visible as methods). There is no get, update, etc. This is on purpose, so that the only visible attributes (those you get by tab-completion for instance) are the those you injected. ```pycon >>> da = AttrDict(foo='bar', life=42) ``` You get the “keys as attributes” that you get with `AttrContainer`: ```pycon >>> da.foo 'bar' ``` But additionally, you get the extra `Mapping` methods: ```pycon >>> list(da.keys()) ['foo', 'life'] >>> list(da.values()) ['bar', 42] >>> da.get('foo') 'bar' >>> da.get('not_a_key', 'default') 'default' ``` You can assign through key or attribute assignment: ```pycon >>> da['true'] = 'love' >>> da.friends = 'forever' >>> list(da.items()) [('foo', 'bar'), ('life', 42), ('true', 'love'), ('friends', 'forever')] ``` etc. #### SEE ALSO Objects in `py2store.utils.attr_dict` module ### *class* dol.sources.Attrs(obj, key_filt=, getattrs=) Bases: [`Store`](_autosummary/dol.base.html.md#dol.base.Store) A simple recursive KvReader for the attributes of a python object. Keys are attr names, values are Attrs(attr_val) instances. #### NOTE A more significant version of Attrs, along with many tools based on it, was moved to pypi package: guide. pip install guide #### update(\*\*F) → None. Update D from mapping/iterable E and F. If E present and has a .keys() method, does: for k in E.keys(): D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v #### update_keys_cache(keys) Updates the \_keys_cache by calling its {} method ### *class* dol.sources.CachedKeysSequenceKvReader(sequence, key=None, val=None, val_postproc=) Bases: [`Store`](_autosummary/dol.base.html.md#dol.base.Store) SequenceKvReader but with keys cached. Use this one if you will perform multiple accesses to only some of the keys of the store #### update(\*\*F) → None. Update D from mapping/iterable E and F. If E present and has a .keys() method, does: for k in E.keys(): D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v #### update_keys_cache(keys) Updates the \_keys_cache by deleting the attribute ### *class* dol.sources.CachedSequenceKvReader(sequence, key=None, val=None, val_postproc=) Bases: [`CachedSequenceKvReader`](_autosummary/dol.sources.html.md#dol.sources.CachedSequenceKvReader) SequenceKvReader but with the whole mapping cached as a dict. Use this one if you will perform multiple accesses to the store ### *class* dol.sources.CascadedStores(stores, default=None, , get_existing_values_only=False, need_to_set_all_stores=False, ignore_non_existing_store_keys=False, \*\*kwargs) Bases: [`Store`](_autosummary/dol.base.html.md#dol.base.Store) A MutableMapping interface to a collection of stores that will write a value in all the stores it contains, read it from the first store it finds that has it, and write it back to all the stores up to the store where it found it. This is useful, for example, when you want to, say, write something to disk, and possibly to a remote backup or shared store, but also keep that value in memory. The name `CascadedStores` comes from “Cascaded Caches”, which is a common pattern in caching systems (e.g. [https://philipwalton.com/articles/cascading-cache-invalidation/](https://philipwalton.com/articles/cascading-cache-invalidation/)) To demo this, let’s create a couple of stores that print when they get a value: ```pycon >>> from collections import UserDict >>> class LoggedDict(UserDict): ... def __init__(self, name: str): ... self.name = name ... super().__init__() ... def __getitem__(self, k): ... print(f"Getting {k} from {self.name}") ... return super().__getitem__(k) >>> cache = LoggedDict('cache') >>> disk = LoggedDict('disk') >>> remote = LoggedDict('remote') ``` Now we can create a CascadedStores instance with these stores and write a value to it: ```pycon >>> stores = CascadedStores([cache, disk, remote]) >>> stores['f'] = 42 ``` See that it’s in both stores: ```pycon >>> cache['f'] Getting f from cache 42 >>> disk['f'] Getting f from disk 42 >>> remote['f'] Getting f from remote 42 ``` See how it reads from the first store only, because it found the `f` key there: ```pycon >>> stores['f'] Getting f from cache 42 ``` Let’s write something in disk only: ```pycon >>> disk['g'] = 43 ``` Now if you ask for `g`, it won’t find it in cache, but will find it in `disk` and return it. ```pycon >>> stores['g'] Getting g from disk 43 ``` Here’s the thing though. Now, `g` is also in `cache`: ```pycon >>> cache {'f': 42, 'g': 43} ``` But `remote` still only has `f`: ```pycon >>> remote {'f': 42} ``` #### from_variadics(\*args, \*\*kwargs) A way to create a fan-out store from a mix of args and kwargs, instead of a single dict. * **Parameters:** * **args** – sub-stores used to fan-out the data. These stores will be represented by their index in the tuple. * **kwargs** – sub-stores used to fan-out the data. These stores will be represented by their name in the dict. \_\_init_\_ arguments can also be passed as kwargs (i.e. `default`, `get_existing_values_only`, and any other subclass specific arguments). Let’s use the same sub-stores: ```pycon >>> bytes_store = dict( ... a=b'a', ... b=b'b', ... c=b'c', ... ) >>> metadata_store = dict( ... b=dict(x=2), ... c=dict(x=3), ... d=dict(x=4), ... ) ``` We can create a fan-out reader from these stores, using args: ```pycon >>> reader = FanoutReader.from_variadics(bytes_store, metadata_store) >>> reader['b'] {0: b'b', 1: {'x': 2}} ``` The reader returns a dict with the values from each store, keyed by the index of the store in the `args` tuple. We can also create a fan-out reader passing the stores in kwargs: ```pycon >>> reader = FanoutReader.from_variadics( ... bytes_store=bytes_store, ... metadata_store=metadata_store ... ) >>> reader['b'] {'bytes_store': b'b', 'metadata_store': {'x': 2}} ``` This way, the returned value is keyed by the name of the store. We can also mix args and kwargs: ```pycon >>> reader = FanoutReader.from_variadics(bytes_store, metadata_store=metadata_store) >>> reader['b'] {0: b'b', 'metadata_store': {'x': 2}} ``` Note that the order of the stores is determined by the order of the args and kwargs. ### dol.sources.Ddir alias of [`Attrs`](_autosummary/dol.sources.html.md#dol.sources.Attrs) ### *class* dol.sources.FanoutPersister(stores, default=None, , get_existing_values_only=False, need_to_set_all_stores=False, ignore_non_existing_store_keys=False, \*\*kwargs) Bases: [`FanoutReader`](_autosummary/dol.sources.html.md#dol.sources.FanoutReader), [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister) A fanout persister is a fanout reader that can also set and delete items. * **Parameters:** * **stores** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – A mapping of store keys to stores. * **default** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The value to return if the key is not in any of the stores. * **get_existing_values_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If True, only return values for stores that contain the key. * **need_to_set_all_stores** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If True, all stores must be set when setting a value. If False, only the stores that are set will be updated. * **ignore_non_existing_store_keys** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If True, ignore store keys from the value that are not in the persister. If False, a ValueError is raised. Let’s create a persister from in-memory stores: ```pycon >>> bytes_store = dict() >>> metadata_store = dict() >>> persister = FanoutPersister( ... stores = dict(bytes_store=bytes_store, metadata_store=metadata_store) ... ) ``` The persister sets the values in each store, based on the store key in the value dict. ```pycon >>> persister['a'] = dict(bytes_store=b'a', metadata_store=dict(x=1)) >>> persister['a'] {'bytes_store': b'a', 'metadata_store': {'x': 1}} ``` By default, not all stores must be set when setting a value: ```pycon >>> persister['b'] = dict(bytes_store=b'b') >>> persister['b'] {'bytes_store': b'b', 'metadata_store': None} ``` This allow to update a subset of the stores whithout having to set all the stores. ```pycon >>> persister['a'] = dict(bytes_store=b'A') >>> persister['a'] {'bytes_store': b'A', 'metadata_store': {'x': 1}} ``` This behavior can be changed by passing `need_to_set_all_stores=True`: ```pycon >>> persister_all_stores = FanoutPersister( ... stores=dict(bytes_store=dict(), metadata_store=dict()), ... need_to_set_all_stores=True, ... ) >>> persister_all_stores['a'] = dict(bytes_store=b'a') Traceback (most recent call last): ... ValueError: All stores must be set when setting a value. Missing stores: {'metadata_store'} ``` By default, if a store key from the value is not in the persister, a ValueError is raised: ```pycon >>> persister['a'] = dict( ... bytes_store=b'a', metadata_store=dict(y=1), other_store='some value' ... ) Traceback (most recent call last): ... ValueError: The value contains some invalid store keys: {'other_store'} ``` This behavior can be changed by passing `ignore_non_existing_store_keys=True`: ```pycon >>> persister_ignore_non_existing_store_keys = FanoutPersister( ... stores=dict(bytes_store=dict(), metadata_store=dict()), ... ignore_non_existing_store_keys=True, ... ) >>> persister_ignore_non_existing_store_keys['a'] = dict( ... bytes_store=b'a', metadata_store=dict(y=1), other_store='some value' ... ) >>> persister_ignore_non_existing_store_keys['a'] {'bytes_store': b'a', 'metadata_store': {'y': 1}} ``` Note that the value of the non-existing store key is ignored! So, be careful when using this option, to avoid losing data. Let’s delete items now: ```pycon >>> del persister['a'] >>> 'a' in persister False ``` The key as been deleted from all the stores: ```pycon >>> 'a' in bytes_store False >>> 'a' in metadata_store False ``` As expected, if the key is not in any of the stores, a KeyError is raised: ```pycon >>> del persister['z'] Traceback (most recent call last): ... KeyError: 'z' ``` However, if the key is in some of the stores, but not in others, the key is deleted from the stores where it is present: ```pycon >>> bytes_store=dict(a=b'a') >>> persister = FanoutPersister( ... stores=dict(bytes_store=bytes_store, metadata_store=dict()), ... ) >>> del persister['a'] >>> 'a' in persister False >>> 'a' in bytes_store False ``` ### *class* dol.sources.FanoutReader(stores, default=None, , get_existing_values_only=False) Bases: [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) Get a ‘fanout view’ of a store of stores. That is, when a key is requested, the key is passed to all the stores, and results accumulated in a dict that is then returned. * **Parameters:** * **stores** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – A mapping of store keys to stores. * **default** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The value to return if the key is not in any of the stores. * **get_existing_values_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If True, only return values for stores that contain the key. Let’s define the following sub-stores: ```pycon >>> bytes_store = dict( ... a=b'a', ... b=b'b', ... c=b'c', ... ) >>> metadata_store = dict( ... b=dict(x=2), ... c=dict(x=3), ... d=dict(x=4), ... ) ``` We can create a fan-out reader from these stores: ```pycon >>> stores = dict(bytes_store=bytes_store, metadata_store=metadata_store) >>> reader = FanoutReader(stores) >>> reader['b'] {'bytes_store': b'b', 'metadata_store': {'x': 2}} ``` The reader returns a dict with the values from each store, keyed by the name of the store. We can also pass a default value to return if the key is not in the store: ```pycon >>> reader = FanoutReader( ... stores=stores, ... default='no value in this store for this key', ... ) >>> reader['a'] {'bytes_store': b'a', 'metadata_store': 'no value in this store for this key'} ``` If the key is not in any of the stores, a KeyError is raised: ```pycon >>> reader['z'] Traceback (most recent call last): ... KeyError: 'z' ``` We can also pass `get_existing_values_only=True` to only return values for stores that contain the key: ```pycon >>> reader = FanoutReader( ... stores=stores, ... get_existing_values_only=True, ... ) >>> reader['a'] {'bytes_store': b'a'} ``` #### *classmethod* from_variadics(\*args, \*\*kwargs) A way to create a fan-out store from a mix of args and kwargs, instead of a single dict. * **Parameters:** * **args** – sub-stores used to fan-out the data. These stores will be represented by their index in the tuple. * **kwargs** – sub-stores used to fan-out the data. These stores will be represented by their name in the dict. \_\_init_\_ arguments can also be passed as kwargs (i.e. `default`, `get_existing_values_only`, and any other subclass specific arguments). Let’s use the same sub-stores: ```pycon >>> bytes_store = dict( ... a=b'a', ... b=b'b', ... c=b'c', ... ) >>> metadata_store = dict( ... b=dict(x=2), ... c=dict(x=3), ... d=dict(x=4), ... ) ``` We can create a fan-out reader from these stores, using args: ```pycon >>> reader = FanoutReader.from_variadics(bytes_store, metadata_store) >>> reader['b'] {0: b'b', 1: {'x': 2}} ``` The reader returns a dict with the values from each store, keyed by the index of the store in the `args` tuple. We can also create a fan-out reader passing the stores in kwargs: ```pycon >>> reader = FanoutReader.from_variadics( ... bytes_store=bytes_store, ... metadata_store=metadata_store ... ) >>> reader['b'] {'bytes_store': b'b', 'metadata_store': {'x': 2}} ``` This way, the returned value is keyed by the name of the store. We can also mix args and kwargs: ```pycon >>> reader = FanoutReader.from_variadics(bytes_store, metadata_store=metadata_store) >>> reader['b'] {0: b'b', 'metadata_store': {'x': 2}} ``` Note that the order of the stores is determined by the order of the args and kwargs. ### *class* dol.sources.FlatReader(readers) Bases: [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) Get a ‘flat view’ of a store of stores. That is, where keys are `(first_level_key, second_level_key)` pairs. This is useful, for instance, to make a union of stores (you’ll get all the values). ```pycon >>> readers = { ... 'fr': {1: 'un', 2: 'deux'}, ... 'it': {1: 'uno', 2: 'due', 3: 'tre'}, ... } >>> s = FlatReader(readers) >>> list(s) [('fr', 1), ('fr', 2), ('it', 1), ('it', 2), ('it', 3)] >>> s[('fr', 1)] 'un' >>> s['it', 2] 'due' ``` ### *class* dol.sources.FuncDag(funcs, \*\*kwargs) Bases: [`FuncReader`](_autosummary/dol.sources.html.md#dol.sources.FuncReader) ### *class* dol.sources.FuncReader(funcs) Bases: [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) Reader that seeds itself from a data fetching function list Uses the function list names as the keys, and their returned value as the values. For example: You have a list of urls that contain the data you want to have access to. You can write functions that bare the names you want to give to each dataset, and have the function fetch the data from the url, extract the data from the response and possibly prepare it (we advise minimally, since you can always transform from the raw source, but the opposite can be impossible). ```pycon >>> def foo(): ... return 'bar' >>> def pi(): ... return 3.14159 >>> s = FuncReader([foo, pi]) >>> list(s) ['foo', 'pi'] >>> s['foo'] 'bar' >>> s['pi'] 3.14159 ``` You might want to give your own names to the functions. You might even have to (because the callable you’re using doesn’t have a `__name__`). In that case, you can specify a `{name: func, ...}` dict instead of a simple iterable. ```pycon >>> s = FuncReader({'FU': foo, 'Pie': pi}) >>> list(s) ['FU', 'Pie'] >>> s['FU'] 'bar' ``` ### *class* dol.sources.MultiSource(\*sources) Bases: [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) A read-only Mapping that composes multiple sources, tried in order. On key lookup, sources are tried left-to-right until one has the key. On iteration, keys are yielded from all sources (deduplicated, order-preserved). This is useful as the `source` argument to [`dol.caching.mk_sourced_store()`](_autosummary/dol.caching.html.md#dol.caching.mk_sourced_store) when you need fallback across multiple read-only backends. ```pycon >>> s1 = {'a': 1, 'b': 2} >>> s2 = {'b': 20, 'c': 3} >>> ms = MultiSource(s1, s2) >>> ms['a'] 1 >>> ms['b'] 2 >>> ms['c'] 3 >>> sorted(ms) ['a', 'b', 'c'] >>> len(ms) 3 >>> 'c' in ms True >>> 'z' in ms False >>> ms['z'] Traceback (most recent call last): ... KeyError: 'z' ``` ### *exception* dol.sources.NotUnique Bases: [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) Raised when an iterator was expected to have only one element, but had more ### *class* dol.sources.ObjReader(\_obj_of_key) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A reader that uses a specified function to get the contents for a given key. ```pycon >>> # define a contents_of_key that reads stuff from a dict >>> data = {'foo': 'bar', 42: "everything"} >>> def read_dict(k): ... return data[k] >>> pr = ObjReader(_obj_of_key=read_dict) >>> pr['foo'] 'bar' >>> pr[42] 'everything' >>> >>> # define contents_of_key that reads stuff from a file given it's path >>> def read_file(path): ... with open(path) as fp: ... return fp.read() >>> pr = ObjReader(_obj_of_key=read_file) >>> file_where_this_code_is = __file__ ``` `file_where_this_code_is` should be the file where this doctest is written, therefore should contain what I just said: ```pycon >>> 'therefore should contain what I just said' in pr[file_where_this_code_is] True ``` ### *class* dol.sources.SequenceKvReader(sequence, key=None, val=None, val_postproc=) Bases: [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) A KvReader that sources itself in an iterable of elements from which keys and values will be extracted and grouped by key. ```pycon >>> docs = [{'_id': 0, 's': 'a', 'n': 1}, ... {'_id': 1, 's': 'b', 'n': 2}, ... {'_id': 2, 's': 'b', 'n': 3}] >>> ``` Out of the box, SequenceKvReader gives you enumerated integer indices as keys, and the sequence items as is, as vals ```pycon >>> s = SequenceKvReader(docs) >>> list(s) [0, 1, 2] >>> s[1] {'_id': 1, 's': 'b', 'n': 2} >>> assert s.get('not_a_key') is None ``` You can make it more interesting by specifying a val function to compute the vals from the sequence elements ```pycon >>> s = SequenceKvReader(docs, val=lambda x: (x['_id'] + x['n']) * x['s']) >>> assert list(s) == [0, 1, 2] # as before >>> list(s.values()) ['a', 'bbb', 'bbbbb'] ``` But where it becomes more useful is when you specify a key as well. SequenceKvReader will then compute the keys with that function, group them, and return as the value, the list of sequence elements that match that key. ```pycon >>> s = SequenceKvReader(docs, ... key=lambda x: x['s'], ... val=lambda x: {k: x[k] for k in x.keys() - {'s'}}) >>> assert list(s) == ['a', 'b'] >>> assert s['a'] == [{'_id': 0, 'n': 1}] >>> assert s['b'] == [{'_id': 1, 'n': 2}, {'_id': 2, 'n': 3}] ``` The cannonical form of key and val is a function, but if you specify a str, int, or iterable thereof, SequenceKvReader will make an itemgetter function from it, for your convenience. ```pycon >>> s = SequenceKvReader(docs, key='_id') >>> assert list(s) == [0, 1, 2] >>> assert s[1] == [{'_id': 1, 's': 'b', 'n': 2}] ``` The `val_postproc` argument is `list` by default, but what if we don’t specify any? Well then you’ll get an unconsumed iterable of matches ```pycon >>> s = SequenceKvReader(docs, key='_id', val_postproc=None) >>> assert isinstance(s[1], Iterable) ``` The `val_postproc` argument specifies what to apply to this iterable of matches. For example, you can specify `val_postproc=next` to simply get the first matched element: ```pycon >>> s = SequenceKvReader(docs, key='_id', val_postproc=next) >>> assert list(s) == [0, 1, 2] >>> assert s[1] == {'_id': 1, 's': 'b', 'n': 2} ``` We got the whole dict there. What if we just want we didn’t want the \_id, which is used by the key, in our val? ```pycon >>> from functools import partial >>> all_but_s = partial(exclusive_subdict, exclude=['s']) >>> s = SequenceKvReader(docs, key='_id', val=all_but_s, val_postproc=next) >>> assert list(s) == [0, 1, 2] >>> assert s[1] == {'_id': 1, 'n': 2} ``` Suppose we want to have the pair of (‘_id’, ‘n’) values as a key, and only ‘s’ as a value… ```pycon >>> s = SequenceKvReader(docs, key=('_id', 'n'), val='s', val_postproc=next) >>> assert list(s) == [(0, 1), (1, 2), (2, 3)] >>> assert s[1, 2] == 'b' ``` But remember that using `val_postproc=next` will only give you the first match as a val. ```pycon >>> s = SequenceKvReader(docs, key='s', val=all_but_s, val_postproc=next) >>> assert list(s) == ['a', 'b'] >>> assert s['a'] == {'_id': 0, 'n': 1} >>> assert s['b'] == {'_id': 1, 'n': 2} # note that only the first match is returned. ``` If you do want to only grab the first match, but want to additionally assert that there is no more than one, you can specify this with `val_postproc=unique_element`: ```pycon >>> s = SequenceKvReader(docs, key='s', val=all_but_s, val_postproc=unique_element) >>> assert s['a'] == {'_id': 0, 'n': 1} >>> # The following should raise an exception since there's more than one match >>> s['b'] Traceback (most recent call last): ... sources.NotUnique: iterator had more than one element ``` # _autosummary/dol.tools.html.md # dol.tools Various tools to add functionality to stores. Main entry points: - `store_aggregate`: aggregate a store’s items into one object (a Markdown text by default) - `confirm_overwrite`: a `wrap_kvs` preset that asks before overwriting a value - `Forest`: a key-value tree view of nested objects ```pycon >>> from dol.tools import store_aggregate >>> print(store_aggregate({'a': 'x', 'b': 'y'})) ## a x ## b y ``` ### Functions | [`ask_user_for_value_when_missing`](_autosummary/dol.tools.html.md#dol.tools.ask_user_for_value_when_missing)([store, ...]) | Wrap a store so if a value is missing when the user asks for it, they will be given a chance to enter the value they want to write. | |--------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| | [`confirm_overwrite`](_autosummary/dol.tools.html.md#dol.tools.confirm_overwrite)(mapping, k, v[, ...]) | A preset function you can use in wrap_kvs to ask the user to confirm if they're writing a value in a key that already has a different value under it. | | [`convert_to_numerical_if_possible`](_autosummary/dol.tools.html.md#dol.tools.convert_to_numerical_if_possible)(s) | To be used with `ask_user_for_value_when_missing` `value_preprocessor` arg | | `decode_as_latin1`(b) | | | `identity`(x) | | | `markdown_section`(k, v) | | | `return_input`(x) | | | `save_string_to_filepath`(filepath, string) | | | [`store_aggregate`](_autosummary/dol.tools.html.md#dol.tools.store_aggregate)(content_store, \*[, ...]) | Create an aggregate object of a store's (a Mapping of strings) content | | `type_check_if_type`(filt) | | ### Classes | [`Forest`](_autosummary/dol.tools.html.md#dol.tools.Forest)(src, \*, get_node_keys, get_src_item, ...) | Provides a key-value forest interface to objects. | |------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------| | `NoSuchKey`() | | | [`iSliceStore`](_autosummary/dol.tools.html.md#dol.tools.iSliceStore)(store) | Wraps a store to make a reader that acts as if the store was a list (with integer keys, and that can be sliced). | ### *class* dol.tools.Forest(src, \*, get_node_keys, get_src_item, is_leaf, forest_type=, leaf_trans=) Bases: [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) Provides a key-value forest interface to objects. A [treehttps://en.wikipedia.org/wiki/Tree_(data_structure)](treehttps://en.wikipedia.org/wiki/Tree_(data_structure)) is a nested data structure. A tree has a root, which is the parent of children, who themselves can be parents of further subtrees, or not; in which case they’re called leafs. For more information, see [wikipediaontreeshttps://en.wikipedia.org/wiki/Tree_(data_structure)](wikipediaontreeshttps://en.wikipedia.org/wiki/Tree_(data_structure)) Here we allow one to construct a tree view of any python object, using a key-value interface to the parent-child relationship. A forest is a collection of trees. Arguably, a dictionnary might not be the most impactful example to show here, since it is naturally a tree (therefore a forest), and naturally key-valued: But it has the advantage of being easy to demo with. Where Forest would really be useful is when you (1) want to give a consistent key-value interface to the many various forms that trees and forest objects come in, or even more so when (2) your object’s tree/forest structure is not obvious, so you need to “extract” that view from it (plus give it a consistent key-value interface, so that you can build an ecosystem of tools around it. Anyway, here’s our dictionary example: ```pycon >>> d = { ... 'apple': { ... 'kind': 'fruit', ... 'types': { ... 'granny': {'color': 'green'}, ... 'fuji': {'color': 'red'} ... }, ... 'tasty': True ... }, ... 'acrobat': { ... 'kind': 'person', ... 'nationality': 'french', ... 'brave': True, ... }, ... 'ball': { ... 'kind': 'toy' ... } ... } ``` Must of the time, you’ll want to curry `Forest` to make an `object_to_forest` constructor for a given class of objects. In the case of dictionaries as the one above, this might look like this: ```pycon >>> from functools import partial >>> a_forest = partial( ... Forest, ... is_leaf=lambda k, v: not isinstance(v, dict), ... get_node_keys=lambda v: [vv for vv in iter(v) if not vv.startswith('b')], ... get_src_item=lambda src, k: src[k] ... ) >>> >>> f = a_forest(d) >>> list(f) ['apple', 'acrobat'] ``` Note that we specified in `get_node_keys``that we didn't want to include items whose keys start with ``b` as valid children. Therefore we don’t have our `'ball'` in the list above. Note below which nodes are themselves `Forests`, and whic are leafs: ```pycon >>> ff = f['apple'] >>> isinstance(ff, Forest) True >>> list(ff) ['kind', 'types', 'tasty'] >>> ff['kind'] 'fruit' >>> fff = ff['types'] >>> isinstance(fff, Forest) True >>> list(fff) ['granny', 'fuji'] ``` ### dol.tools.ask_user_for_value_when_missing(store=None, , value_preprocessor=None, on_missing_msg='No such key was found. You can enter a value for it here or simply hit enter to leave the slot empty', \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Wrap a store so if a value is missing when the user asks for it, they will be given a chance to enter the value they want to write. * **Parameters:** * **store** – The store (instance or class) to wrap * **value_preprocessor** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Function to transform the user value before trying to write it (bearing in mind all user specified values are strings) * **on_missing_msg** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – String that will be displayed to prompt the user to enter a value * **Returns:** ### dol.tools.confirm_overwrite(mapping, k, v, user_input_msg='The key {k} already exists and has value {existing_v}. If you want to overwrite it with {v}, confirm by typing {v} here: ') A preset function you can use in wrap_kvs to ask the user to confirm if they’re writing a value in a key that already has a different value under it. ```pycon >>> from dol.trans import wrap_kvs >>> d = {'a': 'apple', 'b': 'banana'} >>> d = wrap_kvs(d, preset=confirm_overwrite) ``` Overwriting `a` with the same value it already has is fine (not really an over-write): ```pycon >>> d['a'] = 'apple' ``` Creating new values is also fine: ```pycon >>> d['c'] = 'coconut' >>> assert d == {'a': 'apple', 'b': 'banana', 'c': 'coconut'} ``` But if we tried to do `d['a'] = 'alligator'`, we’ll get a user input request: ```default The key a already exists and has value apple. If you want to overwrite it with alligator, confirm by typing alligator here: ``` And we’ll have to type `alligator` and press RETURN to make the write go through. ### dol.tools.convert_to_numerical_if_possible(s) To be used with `ask_user_for_value_when_missing` `value_preprocessor` arg ```pycon >>> convert_to_numerical_if_possible("123") 123 >>> convert_to_numerical_if_possible("123.4") 123.4 >>> convert_to_numerical_if_possible("one") 'one' ``` Border case: The strings “infinity” and “inf” actually convert to a valid float. ```pycon >>> convert_to_numerical_if_possible("infinity") inf ``` ### *class* dol.tools.iSliceStore(store) Bases: [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) Wraps a store to make a reader that acts as if the store was a list (with integer keys, and that can be sliced). I say “list”, but it should be noted that the behavior is more that of range, that outputs an element of the list when keying with an integer, but returns an iterable object (a range) if sliced. Here, a map object is returned when the sliceable store is sliced. ```pycon >>> s = {'foo': 'bar', 'hello': 'world', 'alice': 'bob'} >>> sliceable_s = iSliceStore(s) ``` The read-only functionalities of the underlying mapping are still available: ```pycon >>> list(sliceable_s) ['foo', 'hello', 'alice'] >>> 'hello' in sliceable_s True >>> sliceable_s['hello'] 'world' ``` But now you can get slices as well: ```pycon >>> list(sliceable_s[0:2]) ['bar', 'world'] >>> list(sliceable_s[-2:]) ['world', 'bob'] >>> list(sliceable_s[:-1]) ['bar', 'world'] ``` Now, you can’t do `sliceable_s[1]` because `1` isn’t a valid key. But if you really wanted “item number 1”, you can do: ```pycon >>> next(sliceable_s[1:2]) 'world' ``` Note that `sliceable_s[i:j]` is an iterable that needs to be consumed (here, with list) to actually get the data. If you want your data in a different format, you can use `dol.trans.wrap_kvs` for that. ```pycon >>> from dol import wrap_kvs >>> ss = wrap_kvs(sliceable_s, obj_of_data=list) >>> ss[1:3] ['world', 'bob'] >>> sss = wrap_kvs(sliceable_s, obj_of_data=sorted) >>> sss[1:3] ['bob', 'world'] ``` ### dol.tools.store_aggregate(content_store, \*, kv_to_item=, aggregator=, egress=, key_filter=None, value_filter=None, kv_filter=None, local_store_factory=) Create an aggregate object of a store’s (a Mapping of strings) content The function is written to be able to aggregate the keys and/or values of a store, no matter their type, and concatenate them into an object of arbitrary type. That said, the defaults are setup assuming the store’s keys and values are text, and you want to concatenate them into a single string. This is useful, for example, when you have several files in a folder, and you want to create a single text/markdown file with all the content therein. This function filters content from a given content store, converts the key-value pairs to items (usually text), and (if you specify a filepath as the `egress`) saves the aggregate (text) before returning it. * **Parameters:** * **content_store** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)] | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Path to the folder or dol store to read from. * **kv_to_item** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Item`)]) – Function to convert key-value pairs to an Item (usually a string). * **aggregator** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Item`)]], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Aggregate`)]) – The function that will aggregate the items that `kv_to_item` produces. Defaults to ‘nn’.join. * **egress** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`Aggregate`)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The function that will be called on the aggregate before returning it. Defaults to identity. Note that if you provide a string, the function will save the aggregate text to a file, assuming it is indeed text. * **key_filter** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Optional filter for keys. Defaults to None (no filtering). * **value_filter** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Optional filter for values. Defaults to None (no filtering). * **kv_filter** ([`None`](https://docs.python.org/3/builtins/constants.html#None) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)]], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – Optional filter for key-value pairs. Defaults to None (no filtering). * **local_store_factory** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)]]) – Factory function for the local store, used only if `content_store` is an existing folder path. Defaults to Latin1TextFiles. * **Returns:** Usually the aggregate object, which is usually the concatenated text. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) Normally, you’d specify your content store by specifying a root folder (the function will create a Mapping-view of the contents of the folder for you), or make a content store yourself (a Mapping object providing the key-value pairs). To provide a small example, we’ll take a dict as our content store: ```pycon >>> content_store = { ... 'file1.py': '"""Module docstring."""', ... 'file2.py': 'def foo(): pass', ... 'file3.py': '"""Another docstring."""', ... 'file4.md': 'Markdown content here.', ... 'file5.py': '"""If I mention file5.py, I will be excluded."""', ... } ``` Define the filters: ```pycon >>> key_filter = lambda k: k.endswith('.py') # Only include keys that end with '.py' >>> value_filter = lambda v: v.startswith( ... '"""' ... ) # Only include values that start with """ (marking a module docstring) >>> kv_filter = ( ... lambda kv: kv[0] not in kv[1] ... ) # Exclude key-value pairs where the value mentions the key ``` Call the function with the provided filters and settings ```pycon >>> result = store_aggregate( ... content_store=content_store, # The content_store dict ... kv_to_item="{} -> {}".format, # Format key-value pairs as "key -> value" ... key_filter=key_filter, # Key filter: Include only .py files ... value_filter=value_filter, # Value filter: Include only values starting with """ ... kv_filter=kv_filter, # KV filter: Exclude if value contains the key ... aggregator=', '.join, ... egress='~/test.md' ... ) >>> result 'file1.py -> """Module docstring.""", file3.py -> """Another docstring."""' ``` Here, you got the string as the result. If you want to save it to a file, you can provide the save_filepath argument, and it will save the text to the file, and return the save_filepath to you (which ) Recipe: You can do a lot with the `kv_to_text` argument. For example, if your content store doesn’t have string keys or values, you can always extract whatever information you need from them to produce the text that will represent that item. # _autosummary/dol.trans.html.md # dol.trans Tools to wrap stores with key/value transforms, filters, caches and other layers. A wrap leaves the backend untouched and builds a new class (or instance) around it. The decorators built with `store_decorator` (`wrap_kvs`, `filt_iter`, `cached_keys`, `add_path_access`, …) can be applied to a class, to an instance, or used as a factory (`deco(**params)(store)`). Main entry points: - `wrap_kvs`: key/value transforms (`key_of_id`, `obj_of_data`, codecs, …) - `filt_iter`: restrict a store to a subset of its keys - `cached_keys`: cache the key listing of a slow store - `add_path_access`: read/write nested stores through key paths - `kv_wrap`: wrap using an object that holds `_id_of_key`/`_obj_of_data`-style methods ```pycon >>> from dol.trans import wrap_kvs >>> s = wrap_kvs({}, key_of_id=str.upper, id_of_key=str.lower, obj_of_data=int, data_of_obj=str) >>> s['a'] = 1 >>> s.store # the backend holds the transformed key and value {'a': '1'} >>> list(s), s['A'] (['A'], 1) ``` ### Module Attributes | [`confirm_overwrite`](_autosummary/dol.trans.html.md#dol.trans.confirm_overwrite)(self, k, v) | A ready-to-use `wrap_kvs` `preset` that asks (via the builtin `input`) to confirm before overwriting an existing key with a different value (Issue #13). | |----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| ### Functions | [`add_aliases`](_autosummary/dol.trans.html.md#dol.trans.add_aliases)(obj, \*\*aliases) | A function that wraps the object instance and adds aliases. | |------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| | [`add_decoder`](_autosummary/dol.trans.html.md#dol.trans.add_decoder)([store_cls, decoder, name, ...]) | Add a decoder layer to a store. | | [`add_ipython_key_completions`](_autosummary/dol.trans.html.md#dol.trans.add_ipython_key_completions)(store) | Add tab completion that shows you the keys of the store. | | [`add_missing_key_handling`](_autosummary/dol.trans.html.md#dol.trans.add_missing_key_handling)([store, ...]) | Overrides the `__missing__` method of a store with a custom callback. | | [`add_path_access`](_autosummary/dol.trans.html.md#dol.trans.add_path_access)([store, name, path_type, ...]) | Make nested stores (read/write) accessible through key paths (iterable of keys). | | [`add_path_get`](_autosummary/dol.trans.html.md#dol.trans.add_path_get)([store, name, path_type, ...]) | Make nested stores accessible through key paths. | | [`add_store_method`](_autosummary/dol.trans.html.md#dol.trans.add_store_method)(store, \*, method_func[, ...]) | Add methods to store classes or instances | | [`add_wrapper_method`](_autosummary/dol.trans.html.md#dol.trans.add_wrapper_method)([wrap_cls, method_name]) | Decorator that adds a wrapper method (itself a decorator) to a wrapping class Clear? See `mk_wrapper` function and doctest example if not. | | [`affix_key_codec`](_autosummary/dol.trans.html.md#dol.trans.affix_key_codec)([prefix, suffix]) | A factory that creates a key codec that affixes a prefix and suffix to the key | | [`assert_min_num_of_args`](_autosummary/dol.trans.html.md#dol.trans.assert_min_num_of_args)(func, num_of_args) | Assert that a function can be a store method. | | [`autoviv`](_autosummary/dol.trans.html.md#dol.trans.autoviv)([store]) | Opt-in write-through autovivification for key-paths. | | [`cache_iter`](_autosummary/dol.trans.html.md#dol.trans.cache_iter)([store, keys_cache, ...]) | Make a class that wraps input class's \_\_iter_\_ becomes cached. | | [`cached_keys`](_autosummary/dol.trans.html.md#dol.trans.cached_keys)([store, keys_cache, ...]) | Make a class that wraps input class's \_\_iter_\_ becomes cached. | | [`catch_and_cache_error_keys`](_autosummary/dol.trans.html.md#dol.trans.catch_and_cache_error_keys)([store, ...]) | Store that will cache keys as they're accessed, separating those that raised errors and those that didn't. | | [`condition_function_call`](_autosummary/dol.trans.html.md#dol.trans.condition_function_call)([func, condition, ...]) | Decorator: call `func` only when `condition(*args, **kwargs)` holds, else `callback_if_condition_not_met`. | | [`conditional_data_trans`](_autosummary/dol.trans.html.md#dol.trans.conditional_data_trans)([store, \_\_module_\_, ...]) | Wrap `store` so that `data_trans` is applied to the read values satisfying `condition` (others pass through). | | [`confirm_overwrite`](_autosummary/dol.trans.html.md#dol.trans.confirm_overwrite)(self, k, v) | A ready-to-use `wrap_kvs` `preset` that asks (via the builtin `input`) to confirm before overwriting an existing key with a different value (Issue #13). | | [`constant_output`](_autosummary/dol.trans.html.md#dol.trans.constant_output)([return_val]) | Function that returns a constant value no matter what the inputs are. | | [`disable_delitem`](_autosummary/dol.trans.html.md#dol.trans.disable_delitem)(o) | Replace `o.__delitem__` (if any) with a function raising `ValueError`. | | [`disable_setitem`](_autosummary/dol.trans.html.md#dol.trans.disable_setitem)(o) | Replace `o.__setitem__` (if any) with a function raising `ValueError`. | | [`disallow_overwrites`](_autosummary/dol.trans.html.md#dol.trans.disallow_overwrites)(store, \*[, error_msg, ...]) | Return a subclass of `store` whose `__setitem__` raises `OverWritesNotAllowedError` on existing keys (`store` itself is left untouched). | | [`double_up_as_factory`](_autosummary/dol.trans.html.md#dol.trans.double_up_as_factory)(decorator_func) | Repurpose a decorator both as it's original form, and as a decorator factory. | | [`ensure_clear_method`](_autosummary/dol.trans.html.md#dol.trans.ensure_clear_method)([store, clear_method]) | If obj doesn't have an enabled clear method, will add one (a slow one that runs through keys and deletes them | | [`ensure_set`](_autosummary/dol.trans.html.md#dol.trans.ensure_set)(x) | A set from `x`, treating a string as a single element. | | [`filt_iter`](_autosummary/dol.trans.html.md#dol.trans.filt_iter)([store, filt, name, \_\_module_\_, ...]) | Make a wrapper that will transform a store (class or instance thereof) into a sub-store (i.e. subset of keys). | | [`filter_prefixes`](_autosummary/dol.trans.html.md#dol.trans.filter_prefixes)(prefixes) | Make a filter that returns True if a string starts with one of the given prefixes | | [`filter_regex`](_autosummary/dol.trans.html.md#dol.trans.filter_regex)(regex, \*[, return_search_func]) | Make a filter that returns True if a string matches the given regex | | [`filter_suffixes`](_autosummary/dol.trans.html.md#dol.trans.filter_suffixes)(suffixes) | Make a filter that returns True if a string ends with one of the given suffixes | | [`flatten`](_autosummary/dol.trans.html.md#dol.trans.flatten)([store, levels, cache_keys, ...]) | Give a nested store a flat view whose keys are the `(a, b, c)` key paths. | | [`get_class_name`](_autosummary/dol.trans.html.md#dol.trans.get_class_name)(cls[, dflt_name]) | The `__qualname__` of `cls` (or of its class), else `dflt_name`; raises `ValueError` if there is neither. | | [`ignore_if_error`](_autosummary/dol.trans.html.md#dol.trans.ignore_if_error)([store, errors]) | Wrap `store` so that `__getitem__` errors in `errors` return `None` instead of raising. | | [`insert_aliases`](_autosummary/dol.trans.html.md#dol.trans.insert_aliases)([store, write, read, delete, ...]) | Insert method aliases of CRUD operations of a store (class or instance). | | [`insert_hash_method`](_autosummary/dol.trans.html.md#dol.trans.insert_hash_method)([store, hash_method, ...]) | Make a store hashable using the specified `hash_method`. | | [`insert_load_dump_aliases`](_autosummary/dol.trans.html.md#dol.trans.insert_load_dump_aliases)([store, delete, ...]) | Insert load and dump methods, with familiar dump(obj, location) signature. | | [`is_iterable`](_autosummary/dol.trans.html.md#dol.trans.is_iterable)(x) | Whether `x` is an `Iterable`. | | [`iterate_values_and_accumulate_non_error_keys`](_autosummary/dol.trans.html.md#dol.trans.iterate_values_and_accumulate_non_error_keys)(...) | Yield the values of `store`, appending to `cache_keys_here` the keys whose value was fetched without error. | | [`kv_wrap`](_autosummary/dol.trans.html.md#dol.trans.kv_wrap)(trans_obj) | A function that makes a wrapper (a decorator) that will get the wrappers from methods of the input object. | | [`kv_wrap_persister_cls`](_autosummary/dol.trans.html.md#dol.trans.kv_wrap_persister_cls)(persister_cls[, name]) | Make a class that wraps a persister into a dol.base.Store, | | [`leveled_paths_walk`](_autosummary/dol.trans.html.md#dol.trans.leveled_paths_walk)(m, levels) | Yield the key paths of `m`, down to `levels` levels. | | [`mk_confirm_overwrite_preset`](_autosummary/dol.trans.html.md#dol.trans.mk_confirm_overwrite_preset)(\*[, get_input, ...]) | Make a `wrap_kvs` `preset` that asks for confirmation before overwriting an existing key that holds a *different* value. | | [`mk_kv_reader_from_kv_collection`](_autosummary/dol.trans.html.md#dol.trans.mk_kv_reader_from_kv_collection)(kv_collection) | Make a KvReader class from a Collection class. | | [`mk_level_walk_filt`](_autosummary/dol.trans.html.md#dol.trans.mk_level_walk_filt)(levels) | Makes a `walk_filt` function for `kv_walk` based on some level logic. | | [`mk_read_only`](_autosummary/dol.trans.html.md#dol.trans.mk_read_only)(o) | Disable `__setitem__` and `__delitem__` on `o` (a store class, typically). | | [`mk_trans_obj`](_autosummary/dol.trans.html.md#dol.trans.mk_trans_obj)(\*\*kwargs) | Convenience method to quickly make a trans_obj (just an object holding some trans functions | | [`mk_wrapper`](_autosummary/dol.trans.html.md#dol.trans.mk_wrapper)(wrap_cls) | You have a wrapper class and you want to make a wrapper out of it, that is, a decorator factory with which you can make wrappers, like this: | | [`raise_disabled_error`](_autosummary/dol.trans.html.md#dol.trans.raise_disabled_error)(functionality) | Make a function that raises `ValueError(' is disabled')` whenever called. | | [`redirect_getattr_to_getitem`](_autosummary/dol.trans.html.md#dol.trans.redirect_getattr_to_getitem)([cls, ...]) | A mapping decorator that redirects attribute access to \_\_getitem_\_. | | [`return_default_if_error`](_autosummary/dol.trans.html.md#dol.trans.return_default_if_error)([store, default, errors]) | Wrap `store` so that `__getitem__` errors in `errors` return `default` instead of raising. | | [`store_decorator`](_autosummary/dol.trans.html.md#dol.trans.store_decorator)(func) | Helper to make store decorators. | | [`store_wrap`](_autosummary/dol.trans.html.md#dol.trans.store_wrap)(obj) | Wrap a class or an instance in a `Store` (a class gets a `Store` subclass whose `__init__` builds the wrapped instance). | | [`take_everything`](_autosummary/dol.trans.html.md#dol.trans.take_everything)(key) | Key filter that accepts every key. | | [`transparent_key_method`](_autosummary/dol.trans.html.md#dol.trans.transparent_key_method)(self, k) | Return the key as is (the default `getitem` of `mk_kv_reader_from_kv_collection`). | | [`warn_and_ignore_if_error`](_autosummary/dol.trans.html.md#dol.trans.warn_and_ignore_if_error)([store, errors, ...]) | Like `ignore_if_error`, but also emit a warning (`warn_msg`) for each ignored error. | | [`wrap_kvs`](_autosummary/dol.trans.html.md#dol.trans.wrap_kvs)([store, wrapper, name, key_of_id, ...]) | Make a Store that is wrapped with the given key/val transformers. | ### Classes | [`CachedInvertibleTrans`](_autosummary/dol.trans.html.md#dol.trans.CachedInvertibleTrans)(trans_func) | | |--------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------| | [`Codec`](_autosummary/dol.trans.html.md#dol.trans.Codec)(encoder, decoder) | An `encoder`/`decoder` pair; iterates as `(encoder, decoder)` and composes with `compose_with`. | | [`FiltIter`](_autosummary/dol.trans.html.md#dol.trans.FiltIter)(\*args, \*\*kwargs) | Namespace of `filt_iter` factories (`regex`, `suffixes`, ...); not meant to be instantiated. | | [`FirstArgIsMapping`](_autosummary/dol.trans.html.md#dol.trans.FirstArgIsMapping)(val) | Mark a transform so its first argument is the store (mapping), not the data. | | [`KeyCodec`](_autosummary/dol.trans.html.md#dol.trans.KeyCodec)(encoder, decoder) | A `Codec` that, called on a store, wraps its keys (`id_of_key`/`key_of_id`). | | [`KeyValueCodec`](_autosummary/dol.trans.html.md#dol.trans.KeyValueCodec)(encoder, decoder) | A `Codec` that, called on a store, wraps values with key context (`preset`/`postget`). | | [`OverWritesNotAllowedMixin`](_autosummary/dol.trans.html.md#dol.trans.OverWritesNotAllowedMixin)() | Mixin for only allowing a write to a key if they key doesn't already exist. | | [`SimpleDelegator`](_autosummary/dol.trans.html.md#dol.trans.SimpleDelegator)(obj) | Forward attribute access (and calls) to the wrapped `obj`. | | [`ValueCodec`](_autosummary/dol.trans.html.md#dol.trans.ValueCodec)(encoder, decoder) | A `Codec` that, called on a store, wraps its values (`data_of_obj`/`obj_of_data`). | ### Exceptions | [`MapInvertabilityError`](_autosummary/dol.trans.html.md#dol.trans.MapInvertabilityError) | To be used to indicate that a mapping isn't, or wouldn't be, invertible | |--------------------------------------------------------------------------|---------------------------------------------------------------------------| ### *class* dol.trans.CachedInvertibleTrans(trans_func) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) ```pycon >>> t = CachedInvertibleTrans(lambda x: x[1]) >>> t.ingress('ab') 'b' >>> t.ingress((1, 2)) 2 >>> t.egress('b') 'ab' >>> t.egress(2) (1, 2) ``` ### *class* dol.trans.Codec(encoder, decoder) Bases: [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic)[`DecodedType`, `EncodedType`] An `encoder`/`decoder` pair; iterates as `(encoder, decoder)` and composes with `compose_with`. #### invert() Return a codec that is the inverse of this one. That is, encoder and decoder will be swapped. ### *class* dol.trans.FiltIter(\*args, \*\*kwargs) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Namespace of `filt_iter` factories (`regex`, `suffixes`, …); not meant to be instantiated. #### prefixes() Make a mapping-filtering decorator that filters keys with a prefixes. * **Parameters:** **prefixes** – A string or iterable of strings that are the prefixes to filter ```pycon >>> is_test = filt_iter.prefixes('test') >>> d = {'test.txt': 1, 'report.doc': 2, 'test_image.jpg': 3} >>> dd = is_test(d) >>> dict(dd) {'test.txt': 1, 'test_image.jpg': 3} ``` #### regex() Make a mapping-filtering decorator that filters keys with a regex. * **Parameters:** **regex** – A regex string or compiled regex ```pycon >>> contains_a = filt_iter.regex(r'a') >>> d = {'apple': 1, 'banana': 2, 'cherry': 3} >>> dd = contains_a(d) >>> dict(dd) {'apple': 1, 'banana': 2} ``` #### suffixes() Make a mapping-filtering decorator that filters keys with a suffixes. * **Parameters:** **suffixes** – A string or iterable of strings that are the suffixes to filter ```pycon >>> is_text = filt_iter.suffixes(['.txt', '.doc', '.pdf']) >>> d = {'test.txt': 1, 'report.doc': 2, 'image.jpg': 3} >>> dd = is_text(d) >>> dict(dd) {'test.txt': 1, 'report.doc': 2} ``` ### *class* dol.trans.FirstArgIsMapping(val) Bases: [`LiteralVal`](_autosummary/dol.util.html.md#dol.util.LiteralVal) Mark a transform so its first argument is the store (mapping), not the data. Use this to explicitly opt a transform function into the `f(self, data)` calling convention in wrappers such as `wrap_kvs` – instead of relying on the name/arity heuristic (`_has_unbound_self()`). This is the escape hatch for functions the heuristic can’t (or shouldn’t) infer, and the recommended, explicit alternative to naming a transform’s first parameter `self`/`store`/`mapping`. ```pycon >>> from dol import wrap_kvs, FirstArgIsMapping >>> def prefix_with_name(self, data): ... return f"{getattr(self, 'name', '?')}:{data}" >>> S = wrap_kvs(dict, obj_of_data=FirstArgIsMapping(prefix_with_name)) >>> s = S({'a': 'x'}); s.name = 'ns' >>> s['a'] 'ns:x' ``` ### *class* dol.trans.KeyCodec(encoder, decoder) Bases: [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic)[`DecodedType`, `EncodedType`], [`Codec`](_autosummary/dol.trans.html.md#dol.trans.Codec)[`DecodedType`, `EncodedType`] A `Codec` that, called on a store, wraps its keys (`id_of_key`/`key_of_id`). ### *class* dol.trans.KeyValueCodec(encoder, decoder) Bases: [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic)[`DecodedType`, `EncodedType`], [`Codec`](_autosummary/dol.trans.html.md#dol.trans.Codec)[`DecodedType`, `EncodedType`] A `Codec` that, called on a store, wraps values with key context (`preset`/`postget`). ### *exception* dol.trans.MapInvertabilityError Bases: [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) To be used to indicate that a mapping isn’t, or wouldn’t be, invertible ### *class* dol.trans.OverWritesNotAllowedMixin Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mixin for only allowing a write to a key if they key doesn’t already exist. #### NOTE Should be before the persister in the MRO. ```pycon >>> class TestPersister(OverWritesNotAllowedMixin, dict): ... pass >>> p = TestPersister() >>> p['foo'] = 'bar' >>> #p['foo'] = 'bar2' # will raise error >>> p['foo'] = 'this value should not be stored' Traceback (most recent call last): ... dol.errors.OverWritesNotAllowedError: key foo already exists and cannot be overwritten. If you really want to write to that key, delete it before writing >>> p['foo'] # foo is still bar 'bar' >>> del p['foo'] >>> p['foo'] = 'this value WILL be stored' >>> p['foo'] 'this value WILL be stored' ``` ### *class* dol.trans.SimpleDelegator(obj) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Forward attribute access (and calls) to the wrapped `obj`. ### *class* dol.trans.ValueCodec(encoder, decoder) Bases: [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic)[`DecodedType`, `EncodedType`], [`Codec`](_autosummary/dol.trans.html.md#dol.trans.Codec)[`DecodedType`, `EncodedType`] A `Codec` that, called on a store, wraps its values (`data_of_obj`/`obj_of_data`). ### dol.trans.add_aliases(obj, \*\*aliases) A function that wraps the object instance and adds aliases. See also, and not to be confused with `insert_aliases`, which adds aliases to dunder mapping methods (like `__iter__`, `__getitem__`) etc. ### dol.trans.add_decoder(store_cls=None, , decoder=None, name=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Add a decoder layer to a store. #### NOTE This is a convenience function for `wrap_kvs(..., obj_of_data=decoder)`. ```pycon >>> s = {'a': "42"} >>> ss = add_decoder(s, decoder=int) >>> ss['a'] 42 ``` If there’s only one callable argument, it is assumed to be the decoder: ```pycon >>> wrapper = add_decoder(int) >>> S = wrapper(dict) >>> sss = S({'a': "42"}) >>> dict(sss) == {'a': 42} True ``` ### dol.trans.add_ipython_key_completions(store) Add tab completion that shows you the keys of the store. #### NOTE ipython already adds local path listing automatically, so you’ll still get those along with your valid store keys. ### dol.trans.add_missing_key_handling(store=None, \*, missing_key_callback, errors_that_trigger_missing=(, ), \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Overrides the `__missing__` method of a store with a custom callback. #### NOTE The callback must have two arguments: the store and the key. * **Parameters:** * **store** – The store class to wrap. * **missing_key_callback** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – Function(store, key) -> value for missing keys. * **errors_that_trigger_missing** – Tuple of exceptions that trigger \_\_missing_\_. In the following example, we endow a store to return a sub-store when a key is missing. This substore will contain only keys that start with that missing key. This is useful, for example, to get “subfolder filtering” on a store. ```pycon >>> def prefix_filter(store, prefix: str): ... '''Filter the store to have only keys that start with prefix''' ... from dol import filt_iter ... return filt_iter(store, filt=lambda x: x.startswith(prefix)) ... >>> @add_missing_key_handling(missing_key_callback=prefix_filter) ... class D(dict): ... pass >>> >>> s = D({'a/b': 1, 'a/c': 2, 'd/e': 3, 'f': 4}) >>> sorted(s) ['a/b', 'a/c', 'd/e', 'f'] >>> 'a/' not in s True >>> # yet >>> v = s['a/'] >>> assert dict(v) == {'a/b': 1, 'a/c': 2} ``` ### dol.trans.add_path_access(store=None, \*, name=None, path_type=, create_missing=False, mk_missing=None, explore_further=None, may_create=None, on_create=, max_created=None, max_levels=20, verify_writeback=False, writeback_lock=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make nested stores (read/write) accessible through key paths (iterable of keys). Like `add_path_get`, but with write and delete accessible through key paths. In a way “flatten the nested keys access”. (Warning: `path_type` only effects the first level. That is, it doesn’t work recursively. See issue: [https://github.com/i2mint/dol/issues/10](https://github.com/i2mint/dol/issues/10).) By default, the path object will be a tuple (e.g. `('a', 'b', 'c')`, but you can make it whatever you want, and/or use `dol.paths.KeyPath` to map to and from forms like `'a.b.c'`, `'a/b/c'`, etc. Say you have some nested stores. You know… like a `ZipFileReader` store whose values are `ZipReader` instances, whose values are bytes of the zipped files (and you can go on… whose (json) values are…). For our example, let’s take a nested dict instead: ```pycon >>> s = {'a': {'b': {'c': 42}}} ``` Well, you can access any node of this nested tree of stores like this: ```pycon >>> s['a']['b']['c'] 42 ``` And that’s fine. But maybe you’d like to do it this way instead: ```pycon >>> s = add_path_access(s) >>> s['a', 'b', 'c'] 42 ``` So far, this is what `add_path_get` does. With `add_path_access` though you can also write and delete that way too: ```pycon >>> s['a', 'b', 'c'] = 3.14 >>> s['a', 'b', 'c'] 3.14 >>> del s['a', 'b', 'c'] >>> s {'a': {'b': {}}} ``` You might also want to access 42 with `a.b.c` or `a/b/c` etc. To do that you can use `dol.paths.KeyPath` in combination with * **Parameters:** * **store** – The store (class or instance) you’re wrapping. If not specified, the function will return a decorator. * **name** – The name to give the class (not applicable to instance wrapping) * **path_type** ([`type`](https://docs.python.org/3/builtins/functions.html#type)) – The type that paths are expressed as. Needs to be an Iterable type. By default, a tuple. This is used to decide whether the key should be taken as a “normal” key of the store, or should be used to iterate through, recursively getting values. * **Returns:** A wrapped store (class or instance), or a store wrapping decorator (if store is not specified) #### SEE ALSO `KeyPath` in `dol.paths` Wrapping a class ```pycon >>> S = add_path_access(dict) >>> s = S(a={'b': {'c': 42}}) >>> assert s['a'] == {'b': {'c': 42}}; >>> assert s['a', 'b'] == {'c': 42}; >>> assert s['a', 'b', 'c'] == 42 >>> s['a', 'b', 'c'] = 3.14 >>> s['a', 'b', 'c'] 3.14 >>> del s['a', 'b', 'c'] >>> s {'a': {'b': {}}} ``` Using add_path_get as a decorator ```pycon >>> @add_path_access ... class S(dict): ... pass >>> s = S(a={'b': {'c': 42}}) >>> assert s['a'] == {'b': {'c': 42}}; >>> assert s['a', 'b'] == s['a']['b'] == {'c': 42}; >>> assert s['a', 'b', 'c'] == s['a']['b']['c'] == 42 >>> s['a', 'b', 'c'] = 3.14 >>> s['a', 'b', 'c'] 3.14 >>> del s['a', 'b', 'c'] >>> s {'a': {'b': {}}} ``` A different kind of path? You can choose a different path_type, but sometimes (say both keys and key paths are strings) You need to involve more tools. Like dol.paths.KeyPath… ```pycon >>> from dol.paths import KeyPath >>> from dol.trans import kv_wrap >>> SS = kv_wrap(KeyPath(path_sep='.'))(S) >>> s = SS({'a': {'b': {'c': 42}}}) >>> assert s['a'] == {'b': {'c': 42}}; >>> assert s['a.b'] == s['a']['b']; >>> assert s['a.b.c'] == s['a']['b']['c'] >>> s['a.b.c'] = 3.14 >>> s {'a': {'b': {'c': 3.14}}} >>> del s['a.b.c'] >>> s {'a': {'b': {}}} ``` #### NOTE The add_path_access doesn’t carry on to values. ```pycon >>> s = add_path_access({'a': {'b': {'c': 42}}}) >>> s['a', 'b', 'c'] 42 >>> # but >>> s['a']['b', 'c'] Traceback (most recent call last): ... KeyError: ('b', 'c') ``` That said, ```pycon >>> add_path_access(s['a'])['b', 'c'] 42 ``` The reason why we don’t do this automatically is that it may not always be desirable. If one wanted to though, one could use `wrap_kvs(obj_of_data=...)` to wrap specific values with `add_path_access`. For example, if you wanted to wrap all mappings recursively, you could: ```pycon >>> from typing import Mapping >>> from dol.util import instance_checker >>> add_path_access_if_mapping = conditional_data_trans( ... condition=instance_checker(Mapping), data_trans=add_path_access ... ) >>> s = add_path_access_if_mapping({'a': {'b': {'c': 42}}}) >>> s['a', 'b', 'c'] 42 >>> # But now this works: >>> s['a']['b', 'c'] 42 ``` ### dol.trans.add_path_get(store=None, \*, name=None, path_type=, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make nested stores accessible through key paths. In a way “flatten the nested keys access”. By default, the path object will be a tuple (e.g. `('a', 'b', 'c')`, but you can make it whatever you want, and/or use `dol.paths.KeyPath` to map to and from forms like `'a.b.c'`, `'a/b/c'`, etc. (Warning: `path_type` only effects the first level. That is, it doesn’t work recursively. See issue: [https://github.com/i2mint/dol/issues/10](https://github.com/i2mint/dol/issues/10).) Say you have some nested stores. You know… like a `ZipFileReader` store whose values are `ZipReader` instances, whose values are bytes of the zipped files (and you can go on… whose (json) values are…). For our example, let’s take a nested dict instead: ```pycon >>> s = {'a': {'b': {'c': 42}}} ``` Well, you can access any node of this nested tree of stores like this: ```pycon >>> s['a']['b']['c'] 42 ``` And that’s fine. But maybe you’d like to do it this way instead: ```pycon >>> s = add_path_get(s) >>> s['a', 'b', 'c'] 42 ``` You might also want to access 42 with `a.b.c` or `a/b/c` etc. To do that you can use `dol.paths.KeyPath` in combination with * **Parameters:** * **store** – The store (class or instance) you’re wrapping. If not specified, the function will return a decorator. * **name** – The name to give the class (not applicable to instance wrapping) * **path_type** ([`type`](https://docs.python.org/3/builtins/functions.html#type)) – The type that paths are expressed as. Needs to be an Iterable type. By default, a tuple. This is used to decide whether the key should be taken as a “normal” key of the store, or should be used to iterate through, recursively getting values. * **Returns:** A wrapped store (class or instance), or a store wrapping decorator (if store is not specified) #### SEE ALSO `KeyPath` in `dol.paths` Wrapping an instance ```pycon >>> s = add_path_get({'a': {'b': {'c': 42}}}) >>> s['a'] {'b': {'c': 42}} >>> s['a', 'b'] {'c': 42} >>> s['a', 'b', 'c'] 42 ``` Wrapping a class ```pycon >>> S = add_path_get(dict) >>> s = S(a={'b': {'c': 42}}) >>> assert s['a'] == {'b': {'c': 42}}; >>> assert s['a', 'b'] == {'c': 42}; >>> assert s['a', 'b', 'c'] == 42 ``` Using add_path_get as a decorator ```pycon >>> @add_path_get ... class S(dict): ... pass >>> s = S(a={'b': {'c': 42}}) >>> assert s['a'] == {'b': {'c': 42}}; >>> assert s['a', 'b'] == s['a']['b'] == {'c': 42}; >>> assert s['a', 'b', 'c'] == s['a']['b']['c'] == 42 ``` A different kind of path? You can choose a different path_type, but sometimes (say both keys and key paths are strings) You need to involve more tools. Like dol.paths.KeyPath… ```pycon >>> from dol.paths import KeyPath >>> from dol.trans import kv_wrap >>> SS = kv_wrap(KeyPath(path_sep='.'))(S) >>> s = SS({'a': {'b': {'c': 42}}}) >>> assert s['a'] == {'b': {'c': 42}}; >>> assert s['a.b'] == s['a']['b']; >>> assert s['a.b.c'] == s['a']['b']['c'] ``` ### dol.trans.add_store_method(store, , method_func, method_name=None, validator=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Add methods to store classes or instances * **Parameters:** * **store** ([`type`](https://docs.python.org/3/builtins/functions.html#type)) – A store type or instance * **method_func** – The function of the method to be added * **method_name** – The name of the store attribute this function should be written to * **validator** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`type`](https://docs.python.org/3/builtins/functions.html#type), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – An optional validator. If not None, `validator(store, method_func)` will be called. If it doesn’t return True, a `SetattrNotAllowed` will be raised. Note that `validator` can also raise its own exception. * **Returns:** A store with the added (or modified) method ### dol.trans.add_wrapper_method(wrap_cls=None, , method_name='wrapper') Decorator that adds a wrapper method (itself a decorator) to a wrapping class Clear? See `mk_wrapper` function and doctest example if not. What `add_wrapper_method` does is just to add a `"wrapper"` method (or another name if you ask for it) to `wrap_cls`, so that you can use that class for it’s purpose of transforming stores more conveniently. * **Parameters:** * **wrap_cls** – The wrapper class (the definitioin of the transformation. If None, the functiion will make a decorator to decorate wrap_cls later * **method_name** – The method name you want to use (default is ‘wrapper’) ```pycon >>> >>> @add_wrapper_method ... class RelPath: ... def __init__(self, root): ... self.root = root ... self._root_length = len(root) ... def _key_of_id(self, _id): ... return _id[self._root_length:] ... def _id_of_key(self, k): ... return self.root + k ... >>> RelDict = RelPath.wrapper(root='foo/')(dict) >>> s = RelDict() >>> s['bar'] = 42 >>> assert list(s) == ['bar'] >>> assert s['bar'] == 42 >>> assert str(s) == "{'foo/bar': 42}" # reveals that actually, behind the scenes, there's a "foo/" prefix ``` ### dol.trans.affix_key_codec(prefix='', suffix='') A factory that creates a key codec that affixes a prefix and suffix to the key ```pycon >>> codec = affix_key_codec(prefix='/folder/', suffix='.txt') >>> codec.encoder('name') '/folder/name.txt' >>> codec.decoder('/folder/name.txt') 'name' ``` ### dol.trans.assert_min_num_of_args(func, num_of_args) Assert that a function can be a store method. That is, it should have a signature that takes the store as the first argument ### dol.trans.autoviv(store=None, \*\*kwargs) Opt-in write-through autovivification for key-paths. Shorthand for `add_path_access(store, create_missing=True, **kwargs)`: writing through a key-path (e.g. `s['a', 'b', 'c'] = v`) creates any missing intermediate levels on the way, and the change persists correctly even through persistent / copy-semantics stores (`Files`, `wrap_kvs`-wrapped stores). See `add_path_access` for the full set of options (`mk_missing`, `may_create`, `max_created`, `on_create`, …). Missing-key creation is announced via `warnings.warn` by default, so a typo is never silent. ```pycon >>> from dol import autoviv >>> s = autoviv({}) >>> s['a', 'b', 'c'] = 42 >>> s['a', 'b', 'c'] 42 >>> s['a']['b']['c'] 42 ``` Like `add_path_access`, it also works as a class decorator / factory: ```pycon >>> S = autoviv(dict) >>> s = S() >>> s['x', 'y'] = 1 >>> s['x', 'y'] 1 ``` ### dol.trans.cache_iter(store=None, \*, keys_cache=, iter_to_container=None, cache_update_method='update', name=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make a class that wraps input class’s \_\_iter_\_ becomes cached. Quite often we have a lot of keys, that we get from a remote data source, and don’t want to have to ask for them again and again, having them be fetched, sent over the network, etc. So we need caching. But this caching is not the typical read caching, since it’s \_\_iter_\_ we want to cache, and that’s a generator. So we’ll implement a store class decorator specialized for this. The following decorator, when applied to a class (that has an \_\_iter_\_), will perform the \_\_iter_\_ code, consuming all items of the generator and storing them in \_keys_cache, and then will yield from there every subsequent call. It is assumed, if you’re using the cached_keys transformation, that you’re dealing with static data (or data that can be considered static for the life of the store – for example, when conducting analytics). If you ever need to refresh the cache during the life of the store, you can to delete \_keys_cache like this: ```python del your_store._keys_cache ``` Once you do that, the next time you try to ask something about the contents of the store, it will actually do a live query again, as for the first time. #### NOTE The default keys_cache is list though in many cases, you’d probably should use set, or an explicitly computer set instead. The reason list is used as the default is because (1) we didn’t want to assume that order did not matter (maybe it does to you) and (2) we didn’t want to assume that your keys were hashable. That said, if you’re keys are hashable, and order does not matter, use set. That’ll give you two things: (a) your `key in store` checks will be faster (O(1) instead of O(n)) and (b) you’ll enforce unicity of keys. Know also that if you precompute the keys you want to cache with a container that has an update method (by default `update`) your cache updates will be faster and if the container you use has a `remove` method, you’ll be able to delete as well. * **Parameters:** * **store** – The store instance or class to wrap (must have an \_\_iter_\_), or None if you want a decorator. * **keys_cache** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`Collection`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)) – An explicit collection of keys * **iter_to_container** – The function that will be applied to existing \_\_iter_\_() and assigned to cache. The default is list. Another useful one is the sorted function. * **cache_update_method** – Name of the keys_cache update method to use, if it is an attribute of keys_cache (whether keys_cache is an explicit iterable or a callable). Default `'update'`. * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the new class * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) * **Returns:** If store is None, a decorator that can be applied to a store; if store is a class, a wrapped class that caches its keys; if store is an instance, a wrapped instance that caches its keys. The instances of such key-cached classes have some extra attributes: `_keys_cache` (the actual cache), `_explicit_keys` (whether the cache was given explicitly) and `update_keys_cache` (called on `__setitem__` and `update`). You have two ways of caching keys: - By providing the explicit list of keys you want cache (and use) - By providing a callable that will iterate through your store and collect an explicit list of keys Let’s take a simple dict as our original store. ```pycon >>> source = dict(c=3, b=2, a=1) ``` Specify an iterable, and it will be used as the cached keys ```pycon >>> cached = cached_keys(source, keys_cache='bc') >>> list(cached.items()) # notice that the order you get things is also ruled by the cache [('b', 2), ('c', 3)] ``` Specify a callable, and it will apply it to the existing keys to make your cache ```pycon >>> list(cached_keys(source, keys_cache=sorted)) ['a', 'b', 'c'] ``` You can use the callable keys_cache specification to filter as well! Oh, and let’s demo the fact that if you don’t specify the store, it will make a store decorator for you: ```pycon >>> cache_my_keys = cached_keys(keys_cache=lambda keys: list(filter(lambda k: k >= 'b', keys))) >>> d = cache_my_keys(source) # used as to transform an instance >>> list(d) ['c', 'b'] ``` Let’s use that same `cache_my_keys` to decorate a class instead: ```pycon >>> cached_dict = cache_my_keys(dict) >>> d = cached_dict(c=3, b=2, a=1) >>> list(d) ['c', 'b'] ``` Note that there’s still an underlying store (dict) that has the data: ```pycon >>> repr(d) # repr isn't wrapped, so you can still see your underlying dict "{'c': 3, 'b': 2, 'a': 1}" ``` And yes, you can still add elements, ```pycon >>> d['z'] = 26 >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26)] ``` do bulk updates, ```pycon >>> d.update({'more': 'of this'}, more_of='that') >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26), ('more', 'of this'), ('more_of', 'that')] ``` and delete… ```pycon >>> del d['more'] >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26), ('more_of', 'that')] ``` But careful! Know what you’re doing if you try to get creative. Have a look at this: ```pycon >>> d['a'] = 100 # add an 'a' item >>> d.update(and_more='of that') # update to add yet another item >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26), ('more_of', 'that')] ``` Indeed: No ‘a’ or ‘and_more’. Now… they were indeed added. Or to be more precise, the value of the already existing a was changed, and a new (‘and_more’, ‘of that’) item was indeed added in the underlying store: ```pycon >>> repr(d) "{'c': 3, 'b': 2, 'a': 100, 'z': 26, 'more_of': 'that', 'and_more': 'of that'}" ``` But you’re not seeing it. Why? Because you chose to use a callable keys_cache that doesn’t have an ‘update’ method. When your \_keys_cache attribute (the iterable cache) is not updatable itself, the way updates work is that we iterate through the underlying store (where the updates actually took place), and apply the keys_cache (callable) to that iterable. So what happened here was that you have your new ‘a’ and ‘and_more’ items, but your cached version of the store doesn’t see it because it’s filtered out. On the other hand, check out what happens if you have an updateable cache. Using `set` instead of `list`, after the `filter`. ```pycon >>> cache_my_keys = cached_keys(keys_cache=set) >>> d = cache_my_keys(source) # used as to transform an instance >>> sorted(d) # using sorted because a set's order is not always the same ['a', 'b', 'c'] >>> d['a'] = 100 >>> d.update(and_more='of that') # update to add yet another item >>> sorted(d.items()) [('a', 100), ('and_more', 'of that'), ('b', 2), ('c', 3)] ``` This example was to illustrate a more subtle aspect of cached_keys. You would probably deal with the filter concern in a different way in this case. But the rope is there – it’s your choice on how to use it. And here’s some more examples if that wasn’t enough! ```pycon >>> # Lets cache the keys of a dict. >>> cached_dict = cached_keys(dict) >>> d = cached_dict(a=1, b=2, c=3) >>> # And you get a store that behaves as expected (but more speed and RAM) >>> list(d) ['a', 'b', 'c'] >>> list(d.items()) # whether you iterate with .keys(), .values(), or .items() [('a', 1), ('b', 2), ('c', 3)] ``` This is where the keys are stored: ```pycon >>> d._keys_cache ['a', 'b', 'c'] ``` ```pycon >>> # Let's demo the iter_to_container argument. The default is "list", which will just consume the iter in order >>> sorted_dict = cached_keys(dict, keys_cache=list) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be in the order they were defined ['b', 'a', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=sorted) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be sorted ['a', 'b', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=lambda x: sorted(x, key=len)) >>> s = sorted_dict({'bbb': 3, 'aa': 2, 'c': 1}) >>> list(s) # keys will be sorted according to their length ['c', 'aa', 'bbb'] ``` If you change the keys (adding new ones with \_\_setitem_\_ or update, or removing with pop or popitem) then the cache is recomputed (the first time you use an operation that iterates over keys) ```pycon >>> d.update(d=4) # let's add an element (try d['d'] = 4 as well) >>> list(d) ['a', 'b', 'c', 'd'] >>> d['e'] = 5 >>> list(d.items()) # whether you iterate with .keys(), .values(), or .items() [('a', 1), ('b', 2), ('c', 3), ('d', 4), ('e', 5)] ``` ```pycon >>> @cached_keys ... class A: ... def __iter__(self): ... yield from [1, 2, 3] >>> # Note, could have also used this form: AA = cached_keys(A) >>> a = A() >>> list(a) [1, 2, 3] >>> a._keys_cache = ['a', 'b', 'c'] # changing the cache, to prove that subsequent listing will read from there >>> list(a) # proof: ['a', 'b', 'c'] >>> ``` ```pycon >>> # Let's demo the iter_to_container argument. The default is "list", which will just consume the iter in order >>> sorted_dict = cached_keys(dict, keys_cache=list) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be in the order they were defined ['b', 'a', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=sorted) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be sorted ['a', 'b', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=lambda x: sorted(x, key=len)) >>> s = sorted_dict({'bbb': 3, 'aa': 2, 'c': 1}) >>> list(s) # keys will be sorted according to their length ['c', 'aa', 'bbb'] ``` ### dol.trans.cached_keys(store=None, \*, keys_cache=, iter_to_container=None, cache_update_method='update', name=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make a class that wraps input class’s \_\_iter_\_ becomes cached. Quite often we have a lot of keys, that we get from a remote data source, and don’t want to have to ask for them again and again, having them be fetched, sent over the network, etc. So we need caching. But this caching is not the typical read caching, since it’s \_\_iter_\_ we want to cache, and that’s a generator. So we’ll implement a store class decorator specialized for this. The following decorator, when applied to a class (that has an \_\_iter_\_), will perform the \_\_iter_\_ code, consuming all items of the generator and storing them in \_keys_cache, and then will yield from there every subsequent call. It is assumed, if you’re using the cached_keys transformation, that you’re dealing with static data (or data that can be considered static for the life of the store – for example, when conducting analytics). If you ever need to refresh the cache during the life of the store, you can to delete \_keys_cache like this: ```python del your_store._keys_cache ``` Once you do that, the next time you try to ask something about the contents of the store, it will actually do a live query again, as for the first time. #### NOTE The default keys_cache is list though in many cases, you’d probably should use set, or an explicitly computer set instead. The reason list is used as the default is because (1) we didn’t want to assume that order did not matter (maybe it does to you) and (2) we didn’t want to assume that your keys were hashable. That said, if you’re keys are hashable, and order does not matter, use set. That’ll give you two things: (a) your `key in store` checks will be faster (O(1) instead of O(n)) and (b) you’ll enforce unicity of keys. Know also that if you precompute the keys you want to cache with a container that has an update method (by default `update`) your cache updates will be faster and if the container you use has a `remove` method, you’ll be able to delete as well. * **Parameters:** * **store** – The store instance or class to wrap (must have an \_\_iter_\_), or None if you want a decorator. * **keys_cache** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`Collection`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)) – An explicit collection of keys * **iter_to_container** – The function that will be applied to existing \_\_iter_\_() and assigned to cache. The default is list. Another useful one is the sorted function. * **cache_update_method** – Name of the keys_cache update method to use, if it is an attribute of keys_cache (whether keys_cache is an explicit iterable or a callable). Default `'update'`. * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the new class * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) * **Returns:** If store is None, a decorator that can be applied to a store; if store is a class, a wrapped class that caches its keys; if store is an instance, a wrapped instance that caches its keys. The instances of such key-cached classes have some extra attributes: `_keys_cache` (the actual cache), `_explicit_keys` (whether the cache was given explicitly) and `update_keys_cache` (called on `__setitem__` and `update`). You have two ways of caching keys: - By providing the explicit list of keys you want cache (and use) - By providing a callable that will iterate through your store and collect an explicit list of keys Let’s take a simple dict as our original store. ```pycon >>> source = dict(c=3, b=2, a=1) ``` Specify an iterable, and it will be used as the cached keys ```pycon >>> cached = cached_keys(source, keys_cache='bc') >>> list(cached.items()) # notice that the order you get things is also ruled by the cache [('b', 2), ('c', 3)] ``` Specify a callable, and it will apply it to the existing keys to make your cache ```pycon >>> list(cached_keys(source, keys_cache=sorted)) ['a', 'b', 'c'] ``` You can use the callable keys_cache specification to filter as well! Oh, and let’s demo the fact that if you don’t specify the store, it will make a store decorator for you: ```pycon >>> cache_my_keys = cached_keys(keys_cache=lambda keys: list(filter(lambda k: k >= 'b', keys))) >>> d = cache_my_keys(source) # used as to transform an instance >>> list(d) ['c', 'b'] ``` Let’s use that same `cache_my_keys` to decorate a class instead: ```pycon >>> cached_dict = cache_my_keys(dict) >>> d = cached_dict(c=3, b=2, a=1) >>> list(d) ['c', 'b'] ``` Note that there’s still an underlying store (dict) that has the data: ```pycon >>> repr(d) # repr isn't wrapped, so you can still see your underlying dict "{'c': 3, 'b': 2, 'a': 1}" ``` And yes, you can still add elements, ```pycon >>> d['z'] = 26 >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26)] ``` do bulk updates, ```pycon >>> d.update({'more': 'of this'}, more_of='that') >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26), ('more', 'of this'), ('more_of', 'that')] ``` and delete… ```pycon >>> del d['more'] >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26), ('more_of', 'that')] ``` But careful! Know what you’re doing if you try to get creative. Have a look at this: ```pycon >>> d['a'] = 100 # add an 'a' item >>> d.update(and_more='of that') # update to add yet another item >>> list(d.items()) [('c', 3), ('b', 2), ('z', 26), ('more_of', 'that')] ``` Indeed: No ‘a’ or ‘and_more’. Now… they were indeed added. Or to be more precise, the value of the already existing a was changed, and a new (‘and_more’, ‘of that’) item was indeed added in the underlying store: ```pycon >>> repr(d) "{'c': 3, 'b': 2, 'a': 100, 'z': 26, 'more_of': 'that', 'and_more': 'of that'}" ``` But you’re not seeing it. Why? Because you chose to use a callable keys_cache that doesn’t have an ‘update’ method. When your \_keys_cache attribute (the iterable cache) is not updatable itself, the way updates work is that we iterate through the underlying store (where the updates actually took place), and apply the keys_cache (callable) to that iterable. So what happened here was that you have your new ‘a’ and ‘and_more’ items, but your cached version of the store doesn’t see it because it’s filtered out. On the other hand, check out what happens if you have an updateable cache. Using `set` instead of `list`, after the `filter`. ```pycon >>> cache_my_keys = cached_keys(keys_cache=set) >>> d = cache_my_keys(source) # used as to transform an instance >>> sorted(d) # using sorted because a set's order is not always the same ['a', 'b', 'c'] >>> d['a'] = 100 >>> d.update(and_more='of that') # update to add yet another item >>> sorted(d.items()) [('a', 100), ('and_more', 'of that'), ('b', 2), ('c', 3)] ``` This example was to illustrate a more subtle aspect of cached_keys. You would probably deal with the filter concern in a different way in this case. But the rope is there – it’s your choice on how to use it. And here’s some more examples if that wasn’t enough! ```pycon >>> # Lets cache the keys of a dict. >>> cached_dict = cached_keys(dict) >>> d = cached_dict(a=1, b=2, c=3) >>> # And you get a store that behaves as expected (but more speed and RAM) >>> list(d) ['a', 'b', 'c'] >>> list(d.items()) # whether you iterate with .keys(), .values(), or .items() [('a', 1), ('b', 2), ('c', 3)] ``` This is where the keys are stored: ```pycon >>> d._keys_cache ['a', 'b', 'c'] ``` ```pycon >>> # Let's demo the iter_to_container argument. The default is "list", which will just consume the iter in order >>> sorted_dict = cached_keys(dict, keys_cache=list) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be in the order they were defined ['b', 'a', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=sorted) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be sorted ['a', 'b', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=lambda x: sorted(x, key=len)) >>> s = sorted_dict({'bbb': 3, 'aa': 2, 'c': 1}) >>> list(s) # keys will be sorted according to their length ['c', 'aa', 'bbb'] ``` If you change the keys (adding new ones with \_\_setitem_\_ or update, or removing with pop or popitem) then the cache is recomputed (the first time you use an operation that iterates over keys) ```pycon >>> d.update(d=4) # let's add an element (try d['d'] = 4 as well) >>> list(d) ['a', 'b', 'c', 'd'] >>> d['e'] = 5 >>> list(d.items()) # whether you iterate with .keys(), .values(), or .items() [('a', 1), ('b', 2), ('c', 3), ('d', 4), ('e', 5)] ``` ```pycon >>> @cached_keys ... class A: ... def __iter__(self): ... yield from [1, 2, 3] >>> # Note, could have also used this form: AA = cached_keys(A) >>> a = A() >>> list(a) [1, 2, 3] >>> a._keys_cache = ['a', 'b', 'c'] # changing the cache, to prove that subsequent listing will read from there >>> list(a) # proof: ['a', 'b', 'c'] >>> ``` ```pycon >>> # Let's demo the iter_to_container argument. The default is "list", which will just consume the iter in order >>> sorted_dict = cached_keys(dict, keys_cache=list) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be in the order they were defined ['b', 'a', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=sorted) >>> s = sorted_dict({'b': 3, 'a': 2, 'c': 1}) >>> list(s) # keys will be sorted ['a', 'b', 'c'] >>> sorted_dict = cached_keys(dict, keys_cache=lambda x: sorted(x, key=len)) >>> s = sorted_dict({'bbb': 3, 'aa': 2, 'c': 1}) >>> list(s) # keys will be sorted according to their length ['c', 'aa', 'bbb'] ``` ### dol.trans.catch_and_cache_error_keys(store=None, \*, errors_caught=, error_callback=None, use_cached_keys_after_completed_iter=True, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Store that will cache keys as they’re accessed, separating those that raised errors and those that didn’t. Getting a key will still through an error, but the access attempts will be collected in an ._error_keys attribute. Successfful attemps will be stored in \_keys_cache. Retrieval iteration (items() or values()) will on the other hand, skip the error (while still caching it). If the iteration completes (and use_cached_keys_after_completed_iter), the use_cached_keys flag is turned on, which will result in the store now getting it’s keys from the \_keys_cache. ```pycon >>> @catch_and_cache_error_keys( ... error_callback=lambda store, key, err: print(f"Error with {key} key: {err}")) ... class Blacklist(dict): ... _black_list = {'black', 'list'} ... ... def __getitem__(self, k): ... if k not in self._black_list: ... return super().__getitem__(k) ... else: ... raise KeyError(f"Nope, that's from the black list!") >>> >>> s = Blacklist(black=7, friday=20, frenzy=13) >>> list(s) ['black', 'friday', 'frenzy'] >>> list(s.items()) Error with black key: "Nope, that's from the black list!" [('friday', 20), ('frenzy', 13)] >>> sorted(s) # sorting to get consistent output ['frenzy', 'friday'] ``` See that? First we had three keys, then we iterated and got only 2 items (fortunately, we specified an `error_callback` so we ccould see that the iteration actually dropped a key). That’s strange. And even stranger is the fact that when we list our keys again, we get only two. You don’t like it? Neither do I. But - It’s not a completely outrageous behavior – if you’re talking to live data, it often happens that you get more, or less, from one second to another. - This store isn’t meant to be long living, but rather meant to solve the problem of skiping items that are problematic (for example, malformatted files), with a trace of what was skipped and what’s valid (in case we need to iterate again and don’t want to bear the hit of requesting values for keys we already know are problematic. Here’s a little peep of what is happening under the hood. Meet `_keys_cache` and `_error_keys` sets (yes, unordered – so know it) that are meant to acccumulate valid and problematic keys respectively. ```pycon >>> s = Blacklist(black=7, friday=20, frenzy=13) >>> list(s) ['black', 'friday', 'frenzy'] >>> s._keys_cache, s._error_keys (set(), set()) >>> s['friday'] 20 >>> s._keys_cache, s._error_keys ({'friday'}, set()) >>> s['black'] Traceback (most recent call last): ... KeyError: "Nope, that's from the black list!" >>> s._keys_cache, s._error_keys ({'friday'}, {'black'}) ``` But see that we still have the full list: ```pycon >>> list(s) ['black', 'friday', 'frenzy'] ``` Meet `use_cached_keys`: He’s the culprit. It’s a flag that indicates whether we should be using the cached keys or not. Obviously, it’ll start off being `False`: ```pycon >>> s.use_cached_keys False ``` Now we could set it to `True` manually to change the mode. But know that this switch happens automatically (UNLESS you specify otherwise by saying:`use_cached_keys_after_completed_iter=False`) when ever you got through a VALUE-PRODUCING iteration (i.e. entirely consuming `items()` or `values()`). ```pycon >>> sorted(s.values()) # sorting to get consistent output Error with black key: "Nope, that's from the black list!" [13, 20] ``` ### dol.trans.condition_function_call(func=None, \*, condition=functools.partial(, True), callback_if_condition_not_met=functools.partial(, None)) Decorator: call `func` only when `condition(*args, **kwargs)` holds, else `callback_if_condition_not_met`. ### dol.trans.conditional_data_trans(store=None, , condition, data_trans, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Wrap `store` so that `data_trans` is applied to the read values satisfying `condition` (others pass through). ### dol.trans.confirm_overwrite(self, k, v) A ready-to-use `wrap_kvs` `preset` that asks (via the builtin `input`) to confirm before overwriting an existing key with a different value (Issue #13). For customization (e.g. a non-interactive confirmation policy), use [`mk_confirm_overwrite_preset()`](_autosummary/dol.trans.html.md#dol.trans.mk_confirm_overwrite_preset). ```pycon >>> from dol import wrap_kvs, confirm_overwrite >>> d = wrap_kvs(dict(a='apple', b='banana'), preset=confirm_overwrite) >>> d['a'] = 'apple' # same value -> no prompt, no change >>> d['c'] = 'coconut' # new key -> no prompt, written >>> dict(d) == {'a': 'apple', 'b': 'banana', 'c': 'coconut'} True ``` ### dol.trans.constant_output(return_val=None, \*args, \*\*kwargs) Function that returns a constant value no matter what the inputs are. Is meant to be used with functools.partial to create custom versions. ```pycon >>> from functools import partial >>> always_true = partial(constant_output, True) >>> always_true('regardless', 'of', the='input', will='return True') True ``` ### dol.trans.disable_delitem(o) Replace `o.__delitem__` (if any) with a function raising `ValueError`. Meant for classes: on an instance, `del o[k]` still uses the type’s method. ### dol.trans.disable_setitem(o) Replace `o.__setitem__` (if any) with a function raising `ValueError`. Meant for classes: on an instance, `o[k] = v` still uses the type’s method. ### dol.trans.disallow_overwrites(store, , error_msg=None, disable_deletes=True) Return a subclass of `store` whose `__setitem__` raises `OverWritesNotAllowedError` on existing keys (`store` itself is left untouched). * **Parameters:** * **store** – The store class to wrap (must be a type). * **error_msg** – Custom error message; `{}` (or `{k}`) in it is filled in with the offending key via `.format`. Defaults to a generic message. * **disable_deletes** – If `True` (the default), also disable `__delitem__` (raising the same error) – since deleting a key and rewriting it would otherwise be a way around the overwrite guard. * **Returns:** A new subclass of `store` with the guard(s) attached. ```pycon >>> class D(dict): ... >>> ND = disallow_overwrites(D) >>> d = ND(a=1) >>> d['b'] = 2 >>> d['a'] = 1 Traceback (most recent call last): ... dol.errors.OverWritesNotAllowedError: key a already exists and cannot be overwritten... >>> del d['a'] Traceback (most recent call last): ... dol.errors.OverWritesNotAllowedError: delete of key a is not allowed ``` With `disable_deletes=False`, deletes are left alone: ```pycon >>> ND2 = disallow_overwrites(D, disable_deletes=False) >>> d2 = ND2(a=1) >>> del d2['a'] # no error >>> d2['a'] = 2 # no error either, since 'a' was deleted first ``` ### dol.trans.double_up_as_factory(decorator_func) Repurpose a decorator both as it’s original form, and as a decorator factory. That is, from a decorator that is defined do `wrapped_func = decorator(func, **params)`, make it also be able to do `wrapped_func = decorator(**params)(func)`. #### NOTE You’ll only be able to do this if all but the first argument are keyword-only, and the first argument (the function to decorate) has a default of `None` (this is for your own good). This is validated before making the “double up as factory” decorator. ```pycon >>> @double_up_as_factory ... def decorator(func=None, *, multiplier=2): ... def _func(x): ... return func(x) * multiplier ... return _func ... >>> def foo(x): ... return x + 1 ... >>> foo(2) 3 >>> wrapped_foo = decorator(foo, multiplier=10) >>> wrapped_foo(2) 30 >>> >>> multiply_by_3 = decorator(multiplier=3) >>> wrapped_foo = multiply_by_3(foo) >>> wrapped_foo(2) 9 >>> >>> @decorator(multiplier=3) ... def foo(x): ... return x + 1 ... >>> foo(2) 9 ``` Note that to be able to use double_up_as_factory, your first argument (the object to be wrapped) needs to default to None and be the only argument that is not keyword-only (i.e. all other arguments need to be keyword only). ```pycon >>> @double_up_as_factory ... def decorator_2(func, *, multiplier=2): ... '''Should not be able to be transformed with double_up_as_factory''' Traceback (most recent call last): ... AssertionError: First argument of the decorator function needs to default to None. Was >>> @double_up_as_factory ... def decorator_3(func=None, multiplier=2): ... '''Should not be able to be transformed with double_up_as_factory''' Traceback (most recent call last): ... AssertionError: All arguments (besides the first) need to be keyword-only ``` ### dol.trans.ensure_clear_method(store=None, \*, clear_method=) If obj doesn’t have an enabled clear method, will add one (a slow one that runs through keys and deletes them ### dol.trans.ensure_set(x) A set from `x`, treating a string as a single element. ### dol.trans.filt_iter(store=None, \*, filt=, name=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make a wrapper that will transform a store (class or instance thereof) into a sub-store (i.e. subset of keys). * **Parameters:** * **filt** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)) – A callable or iterable. If a callable, a boolean filter function taking a key and returning True iff the key should be included. If an iterable, the collection of keys you want to filter “in”. * **name** – The name to give the wrapped class * **Returns:** A wrapper (that then needs to be applied to a store instance or class. ```pycon >>> filtered_dict = filt_iter(filt=lambda k: (len(k) % 2) == 1)(dict) # keep only odd length keys >>> >>> s = filtered_dict({'a': 1, 'bb': object, 'ccc': 'a string', 'dddd': [1, 2]}) >>> >>> list(s) ['a', 'ccc'] >>> 'a' in s # True because odd (length) key True >>> 'bb' in s # False because odd (length) key False >>> assert s.get('bb', None) == None >>> len(s) 2 >>> list(s.keys()) ['a', 'ccc'] >>> list(s.values()) [1, 'a string'] >>> list(s.items()) [('a', 1), ('ccc', 'a string')] >>> s.get('a') 1 >>> assert s.get('bb') is None >>> s['x'] = 10 >>> list(s.items()) [('a', 1), ('ccc', 'a string'), ('x', 10)] >>> try: ... s['xx'] = 'not an odd key' ... raise ValueError("This should have failed") ... except KeyError: ... pass ``` ### dol.trans.filter_prefixes(prefixes) Make a filter that returns True if a string starts with one of the given prefixes ```pycon >>> starts_with_test = filter_prefixes('test') >>> starts_with_test("test.txt") True >>> starts_with_test("report.doc") False >>> is_test_or_report = filter_prefixes(['test', 'report']) >>> is_test_or_report("test.txt") True >>> is_test_or_report("report.doc") True >>> is_test_or_report("image.jpg") False ``` The prefixes are grouped, so a multi-prefix filter doesn’t accidentally match a string that merely *contains* one of the later prefixes anywhere: ```pycon >>> is_logs_or_tmp = filter_prefixes(['logs/', 'tmp/']) >>> is_logs_or_tmp("other/tmp/c") False ``` ### dol.trans.filter_regex(regex, , return_search_func=False) Make a filter that returns True if a string matches the given regex ```pycon >>> is_txt = filter_regex(r'.*\.txt') >>> is_txt("test.txt") True >>> is_txt("report.doc") False ``` The argument is a *regular expression*, so it is compiled with `re.compile` – NOT `safe_compile` (which is for file-path templates and `re.escape``s its input on Windows). Using ``safe_compile` here silently broke every regex filter on Windows: e.g. `filter_suffixes('.json')` (used by `Jsons`) had its `(\.json)$` pattern escaped into a literal string matcher, so no `*.json` key matched and the store raised `KeyError: 'Key not in store: .json'`. ```pycon >>> is_json = filter_regex(r"(\.json)$") # works identically on every OS >>> is_json("doc-001.json") True >>> is_json("doc-001.txt") False ``` ### dol.trans.filter_suffixes(suffixes) Make a filter that returns True if a string ends with one of the given suffixes ```pycon >>> ends_with_txt = filter_suffixes('.txt') >>> ends_with_txt("test.txt") True >>> ends_with_txt("report.doc") False >>> is_text = filter_suffixes(['.txt', '.doc', '.pdf']) >>> is_text("test.txt") True >>> is_text("report.doc") True >>> is_text("image.jpg") False ``` ### dol.trans.flatten(store=None, , levels=None, cache_keys=False, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Give a nested store a flat view whose keys are the `(a, b, c)` key paths. Say you have a store that has three levels (or more), that is, that you can always ask for the value `store[a][b][c]` if `a` is a valid key of `store`, `b` is a valid key of `store[a]` and `c` is a valid key of `store[a][b]`. What `flattened_store = flatten(store, levels=3)` will give you is the ability to access the `store[a][b][c]` as `store[a, b, c]`, while still being able to access these stores “normally”. If that’s all you need, you can just use the `add_get_path` wrapper for this. Why would you use `flatten`? Because `add_get_path(store)` would still only give you the `KvReader` point of view of the root `store`. If you `list(store)`, you’d only get the first level keys, or if you ask if `(a, b, c)` is in the store, it will tell you it’s not (though you can access data with such a key. Instead, a flattened store will consider that the keys are those `(a, b, c)` key paths. Further, when flattening a store, you can ask for the view to cache the keys, specifying `cache_keys=True` or give it an explicit place to cache or factory to make a cache (see `cached_keys` wrapper for more details). Though caching keys is not the default it’s highly recommended to do so in most cases. The only reason it is not the default is because if you have millions of keys, but little memory, that’s not what you might want. #### NOTE Flattening just provides a wrapper giving you a “flattened view”. It doesn’t change the store itself, or it’s contents. * **Parameters:** * **store** – The store instance or class to be wrapped * **levels** – The number of nested levels to flatten * **cache_keys** – Whether to cache the keys, or a cache factory or instance. ```pycon >>> from dol import flatten >>> d = { ... 'a': {'b': {'c': 42}}, ... 'aa': {'bb': {'cc': 'dragon_con'}} ... } ``` You can get a flattened view of an instance: ```pycon >>> m = flatten(d, levels=3, cache_keys=True) >>> assert ( ... list(m.items()) ... == [ ... (('a', 'b', 'c'), 42), ... (('aa', 'bb', 'cc'), 'dragon_con') ... ] ... ) ``` You can make a flattener and apply it to an instance (or a class): ```pycon >>> my_flattener = flatten(levels=2) >>> m = my_flattener(d) >>> assert ( ... list(m.items()) ... == [ ... (('a', 'b'), {'c': 42}), ... (('aa', 'bb'), {'cc': 'dragon_con'}) ... ] ... ) ``` Finally, you can wrap a class itself. ```pycon >>> @flatten(levels=1) ... class MyFlatDict(dict): ... pass >>> m = MyFlatDict(d) >>> assert ( ... list(m.items()) ... == [ ... (('a',), {'b': {'c': 42}}), ... (('aa',), {'bb': {'cc': 'dragon_con'}}) ... ] ... ) ``` ### dol.trans.get_class_name(cls, dflt_name=None) The `__qualname__` of `cls` (or of its class), else `dflt_name`; raises `ValueError` if there is neither. ### dol.trans.ignore_if_error(store=None, \*, errors=(, )) Wrap `store` so that `__getitem__` errors in `errors` return `None` instead of raising. ### dol.trans.insert_aliases(store=None, , write=None, read=None, delete=None, list=None, count=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Insert method aliases of CRUD operations of a store (class or instance). If store is a class, you’ll get a copy of the class with those methods added. If store is an instance, the methods will be added in place (no copy will be made). #### NOTE If an operation (write, read, delete, list, count) is not specified, no alias will be created for that operation. IMPORTANT NOTE: The signatures of the methods the aliases will point to will not change. We say this because, you can call the write method “dump”, but you’ll have to use it as `store.dump(key, val)`, not `store.dump(val, key)`, which is the signature you’re probably used to (it’s the one used by json.dump or pickle.dump for example). If you want that familiar interface, using the insert_load_dump_aliases function. See also (and not to be confused with): `add_aliases` * **Parameters:** * **store** – The store to extend with aliases. * **write** – Desired method name for \_\_setitem_\_ * **read** – Desired method name for \_\_getitem_\_ * **delete** – Desired method name for \_\_delitem_\_ * **list** – Desired method name for \_\_iter_\_ * **count** – Desired method name for \_\_len_\_ * **Returns:** A store with the desired aliases. ```pycon >>> # Example of extending a class >>> mydict = insert_aliases(dict, write='dump', read='load', delete='rm', list='peek', count='size') >>> s = mydict(true='love') >>> s.dump('friends', 'forever') >>> s {'true': 'love', 'friends': 'forever'} >>> s.load('true') 'love' >>> list(s.peek()) ['true', 'friends'] >>> s.size() 2 >>> s.rm('true') >>> s {'friends': 'forever'} >>> >>> # Example of extending an instance >>> from collections import UserDict >>> s = UserDict(true='love') # make (and instance) of a UserDict (can't modify a dict instance) >>> # make aliases of note that you don't need >>> s = insert_aliases(s, write='put', read='retrieve', count='num_of_items') >>> s.put('friends', 'forever') >>> s {'true': 'love', 'friends': 'forever'} >>> s.retrieve('true') 'love' >>> s.num_of_items() 2 ``` ### dol.trans.insert_hash_method(store=None, \*, hash_method=, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make a store hashable using the specified `hash_method`. Will add (or overwrite) a `__hash__` method to the store that uses the hash_method to hash the store and an `__eq__` method that compares the store to another based on the hash_method. The `hash_method`, which will be used as the `__hash__` method of a class should return an integer value that represents the hash of the object. To remain sane, the hash value must be the same for an object every time the `__hash__` method is called during the lifetime of the object, and objects that compare equal (using the \_\_eq_\_ method) must have the same hash value. It’s also important that the hash function has the property of being deterministic and returning a hash value that is uniformly distributed across the range of possible integers for the given data. This is important for the hash table data structure to work efficiently. See [This issue](https://github.com/i2mint/dol/issues/7) for further information. ```pycon >>> d = {1: 2} # not hashable! >>> dd = insert_hash_method(d) >>> assert isinstance(hash(dd), int) # now hashable! ``` It looks the same: ```pycon >>> dd {1: 2} ``` But don’t be fooled: dd is not equal to the original `d` (since insert_hash_method\`\`overwrote the `__eq__` method to compare based on the hash value): ```pycon >>> d == dd False ``` But if you cast both to dicts and then compare, you’ll be using the key and value based comparison of dicts, which makes these two equal. ```pycon >>> dict(d) == dict(dd) True ``` The default `hash_method` is `id`, so two hashable wrappers won’t be equal to eachother: ```pycon >>> insert_hash_method(d) == insert_hash_method(d) False ``` In the following we show two things: That you can specify your own custom `hash_method`, and that you can use `insert_hash_method` to wrap classes ```pycon >>> class D(dict): ... pass >>> DD = insert_hash_method(D, hash_method=lambda x: 42) >>> hash(DD(d)) 42 ``` You can also use it as a decorator, without arguments, ```pycon >>> @insert_hash_method ... class E(dict): ... pass >>> assert isinstance(hash(E({1: 2})), int) ``` or with arguments (which you must specify as keyword arguments): ```pycon >>> @insert_hash_method(hash_method=lambda x: sum(x.values())) ... class F(dict): ... pass >>> hash(F({1: 2, 3: 4})) 6 ``` ### dol.trans.insert_load_dump_aliases(store=None, , delete=None, list=None, count=None, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Insert load and dump methods, with familiar dump(obj, location) signature. * **Parameters:** * **store** – The store to extend with aliases. * **delete** – Desired method name for \_\_delitem_\_ * **list** – Desired method name for \_\_iter_\_ * **count** – Desired method name for \_\_len_\_ * **Returns:** A store with the desired aliases. ```pycon >>> mydict = insert_load_dump_aliases(dict) >>> s = mydict() >>> s.dump(obj='love', key='true') >>> s {'true': 'love'} ``` ### dol.trans.is_iterable(x) Whether `x` is an `Iterable`. ### dol.trans.iterate_values_and_accumulate_non_error_keys(store, cache_keys_here, errors_caught=, error_callback=None) Yield the values of `store`, appending to `cache_keys_here` the keys whose value was fetched without error. ### dol.trans.kv_wrap(trans_obj) A function that makes a wrapper (a decorator) that will get the wrappers from methods of the input object. * **Parameters:** **trans_obj** – An object that contains (as attributes) the collection of transformation functions. The attribute names that are used, natively, to make the wrapper are `_key_of_id`, `_id_of_key`, `_obj_of_data`, `_data_of_obj`, `_preset`, and `_postget`. If your `trans_obj` uses different names for these functions, you can use the `add_aliases` function. We’ll demo the use of `add_aliases` here: ```pycon >>> from dol import kv_wrap, add_aliases, Pipe >>> from functools import partial >>> >>> class SeparatorTrans: ... def __init__(self, sep: str): ... self.sep = sep ... def string_to_tuple(self, string: str): ... return tuple(string.split(self.sep)) ... def tuple_to_string(self, tup: tuple): ... return self.sep.join(tup) >>> >>> _add_aliases = partial( ... add_aliases, _key_of_id='string_to_tuple', _id_of_key='tuple_to_string' ... ) >>> mk_sep_trans = Pipe(SeparatorTrans, _add_aliases, kv_wrap) >>> sep_trans = mk_sep_trans('/') >>> d = sep_trans({'a/b/c': 1, 'd/e': 2}) >>> list(d) [('a', 'b', 'c'), ('d', 'e')] >>> d['d', 'e'] 2 ``` `kv_wrap` also has convenience attributes (`outcoming_keys`, `ingoing_keys`, `outcoming_vals`, `ingoing_vals`, and `val_reads_wrt_to_keys`) which will only add a single specific wrapper (specified as a function), when that’s what you need. ### dol.trans.kv_wrap_persister_cls(persister_cls, name=None) Make a class that wraps a persister into a dol.base.Store, * **Parameters:** **persister_cls** – The persister class to wrap * **Returns:** A Store wrapping the persister (see dol.base) ```pycon >>> A = kv_wrap_persister_cls(dict) >>> a = A() >>> a['one'] = 1 >>> a['two'] = 2 >>> a['three'] = 3 >>> list(a.items()) [('one', 1), ('two', 2), ('three', 3)] >>> assert hasattr(a, '_obj_of_data') # for example, it has this magic method >>> # If you overwrite the _obj_of_data method, you'll transform outcomming values with it. >>> # For example, say the data you stored were minutes, but you want to get then in secs... >>> a._obj_of_data = lambda data: data * 60 >>> list(a.items()) [('one', 60), ('two', 120), ('three', 180)] >>> >>> # And if you want to have class that has this weird "store minutes, retrieve seconds", you can do this: >>> class B(kv_wrap_persister_cls(dict)): ... def _obj_of_data(self, data): ... return data * 60 >>> b = B() >>> b.update({'one': 1, 'two': 2, 'three': 3}) # you can write several key-value pairs at once this way! >>> list(b.items()) [('one', 60), ('two', 120), ('three', 180)] >>> # Warning! Advanced under-the-hood chat coming up.... Note this: >>> print(b) {'one': 1, 'two': 2, 'three': 3} >>> # What?!? Well, remember, printing an object calls the objects __str__, which usually calls __repr__ >>> # The wrapper doesn't wrap those methods, since they don't have consistent behaviors. >>> # Here you're getting the __repr__ of the underlying dict store, without the key and value transforms. >>> >>> # Say you wanted to transform the incoming minute-unit data, converting to secs BEFORE they were stored... >>> class C(kv_wrap_persister_cls(dict)): ... def _data_of_obj(self, obj): ... return obj * 60 >>> c = C() >>> c.update(one=1, two=2, three=3) # yet another way you can write multiple key-vals at once >>> list(c.items()) [('one', 60), ('two', 120), ('three', 180)] >>> print(c) # but notice that unlike when we printed b, here the stored data is actually transformed! {'one': 60, 'two': 120, 'three': 180} >>> >>> # Now, just to demonstrate key transformation, let's say that we need internal (stored) keys to be upper case, >>> # but external (the keys you see when listed) ones to be lower case, for some reason... >>> class D(kv_wrap_persister_cls(dict)): ... _data_of_obj = staticmethod(lambda obj: obj * 60) # to demonstrated another way of doing this ... _key_of_id = lambda self, _id: _id.lower() # note if you don't specify staticmethod, 1st arg must be self ... def _id_of_key(self, k): # a function definition like you're used to ... return k.upper() >>> d = D() >>> d['oNe'] = 1 >>> d.update(TwO=2, tHrEE=3) >>> list(d.items()) # you see clean lower cased keys at the interface of the store [('one', 60), ('two', 120), ('three', 180)] >>> # but internally, the keys are all upper case >>> print(d) # equivalent to print(d.store), so keys and values not wrapped (values were transformed before stored) {'ONE': 60, 'TWO': 120, 'THREE': 180} >>> >>> # On the other hand, careful, if you gave the data directly to D, you wouldn't get that. >>> d = D({'one': 1, 'two': 2, 'three': 3}) >>> print(d) {'one': 1, 'two': 2, 'three': 3} >>> # Thus is because when you construct a D with the dict, it initializes the dicts data with it directly >>> # before the key/val transformers are in place to do their jobs. ``` ### dol.trans.leveled_paths_walk(m, levels) Yield the key paths of `m`, down to `levels` levels. ### dol.trans.mk_confirm_overwrite_preset(\*, get_input=None, prompt=) Make a `wrap_kvs` `preset` that asks for confirmation before overwriting an existing key that holds a *different* value. The returned preset writes `v` unchanged unless `k` already maps to a different value; in that case it asks `get_input` to confirm (by typing the new value), and keeps the existing value if the confirmation doesn’t match. Use `get_input` to customize how confirmation is obtained (`None` -> the builtin `input`, resolved at call time), e.g. for testing or for a non-interactive policy. See [`confirm_overwrite()`](_autosummary/dol.trans.html.md#dol.trans.confirm_overwrite). ```pycon >>> store = dict(a='apple', b='banana') >>> # simulate a user who always types the exact new value (confirms every overwrite) >>> always_yes = wrap_kvs( ... store, preset=mk_confirm_overwrite_preset(get_input=lambda prompt: 'alligator') ... ) >>> always_yes['a'] = 'alligator' # confirmation matches -> overwrites >>> store['a'] 'alligator' >>> # simulate a user who declines (types something that doesn't match) >>> always_no = wrap_kvs( ... store, preset=mk_confirm_overwrite_preset(get_input=lambda prompt: 'nope') ... ) >>> always_no['a'] = 'anteater' # declined -> existing value kept >>> store['a'] 'alligator' ``` ### dol.trans.mk_kv_reader_from_kv_collection(kv_collection, name=None, getitem=) Make a KvReader class from a Collection class. * **Parameters:** * **kv_collection** – The Collection class * **name** – The name to give the KvReader class (by default, it will be kv_collection._\_qualname_\_ + ‘Reader’) * **getitem** – The method that will be assigned to \_\_getitem_\_. Should have the (self, k) signature. By default, getitem will be transparent_key_method, returning the key as is. This default is useful when you want to delegate the actual getting to a \_obj_of_data wrapper. * **Returns:** A KvReader class that subclasses the input kv_collection ### dol.trans.mk_level_walk_filt(levels) Makes a `walk_filt` function for `kv_walk` based on some level logic. If `levels` is an integer, will consider it as the max path length, if not it will just assert that `levels` is callable, and return it ### dol.trans.mk_read_only(o) Disable `__setitem__` and `__delitem__` on `o` (a store class, typically). ```pycon >>> class D(dict): ... pass >>> D = mk_read_only(D) >>> D()['a'] = 1 Traceback (most recent call last): ... ValueError: writing is disabled ``` ### dol.trans.mk_trans_obj(\*\*kwargs) Convenience method to quickly make a trans_obj (just an object holding some trans functions ### dol.trans.mk_wrapper(wrap_cls) You have a wrapper class and you want to make a wrapper out of it, that is, a decorator factory with which you can make wrappers, like this: ```python wrapper = mk_wrapper(wrap_cls) ``` that you can then use to transform stores like thiis: ```python MyStore = wrapper(**wrapper_kwargs)(StoreYouWantToTransform) ``` * **Parameters:** **wrap_cls** * **Returns:** ```pycon >>> class RelPath: ... def __init__(self, root): ... self.root = root ... self._root_length = len(root) ... def _key_of_id(self, _id): ... return _id[self._root_length:] ... def _id_of_key(self, k): ... return self.root + k >>> relpath_wrap = mk_wrapper(RelPath) >>> RelDict = relpath_wrap(root='foo/')(dict) >>> s = RelDict() >>> s['bar'] = 42 >>> assert list(s) == ['bar'] >>> assert s['bar'] == 42 >>> assert str(s) == "{'foo/bar': 42}" # reveals that actually, behind the scenes, there's a "foo/" prefix ``` ### dol.trans.raise_disabled_error(functionality) Make a function that raises `ValueError(' is disabled')` whenever called. ### dol.trans.redirect_getattr_to_getitem(cls=None, , keys_have_priority_over_attributes=False, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) A mapping decorator that redirects attribute access to \_\_getitem_\_. #### WARNING This decorator will make your class un-pickleable. * **Parameters:** **keys_have_priority_over_attributes** – If True, keys will have priority over existing attributes. ```pycon >>> @redirect_getattr_to_getitem ... class MyDict(dict): ... pass >>> d = MyDict(a=1, b=2) >>> d.a 1 >>> d.b 2 >>> list(d) ['a', 'b'] ``` ### dol.trans.return_default_if_error(store=None, \*, default=None, errors=(, )) Wrap `store` so that `__getitem__` errors in `errors` return `default` instead of raising. ### dol.trans.store_decorator(func) Helper to make store decorators. You provide a class-decorating function `func` that takes a store type (and possibly additional params) and returns another decorated store type. `store_decorator` takes that `func` and provides an enhanced class decorator specialized for stores. Namely it will: - Add `__module__`, `__qualname__`, `__name__` and `__doc__` arguments to it - Copy the aforementioned arguments to the decorated class, or copy the attributes of the original if not specified. - Output a decorator that can be used in four different ways: a class/instance decorator/factory. By class/instance decorator/factory we mean that if `A` is a class, `a` an instance of it, and `deco` a decorator obtained with `store_decorator(func)`, we can use `deco` to - class decorator: decorate a class - class decorator factory: make a function that decorates classes - instance decorator: decorate an instance of a store - instancce decorator factor: make a function that decorates instances of stores For example, say we have the following `deco` that we made with `store_decorator`: ```pycon >>> @store_decorator ... def deco(cls=None, *, x=1): ... # do stuff to cls, or a copy of it... ... cls.x = x # like this for example ... return cls ``` And a class that has nothing to it: ```pycon >>> class A: ... ``` Nammely, it doesn’t have an `x` ```pycon >>> hasattr(A, 'x') False ``` We make a `decorated_A` with `deco` (class decorator example) ```pycon >>> t = deco(A, x=42) >>> assert isinstance(t, type) ``` and we see that we now have an `x` and it’s 42 ```pycon >>> hasattr(A, 'x') True >>> A.x 42 ``` But we could have also made a factory to decorate `A` and anything else that comes our way. ```pycon >>> paint_it_42 = deco(x=42) >>> decorated_A = paint_it_42(A) >>> assert decorated_A.x == 42 >>> class B: ... x = 'destined to disappear' >>> assert paint_it_42(B).x == 42 ``` To be fair though, you’ll probably see the factory usage appear in the following form, where the class is decorated at definition time. ```pycon >>> @deco(x=42) ... class B: ... pass >>> assert B.x == 42 ``` If your exists already, and you want to keep it as is (with the same name), you can use subclassing to transform a copy of `A` instead, as below. Also note in the following example, that `deco` was used without parentheses, which is equivalent to `@deco()`, and yes, store_decorator makes that possible to, as long as your params have defaults ```pycon >>> @deco ... class decorated_A(A): ... pass >>> assert decorated_A.x == 1 >>> assert A.x == 42 ``` Finally, you can also decorate instances: ```pycon >>> class A: ... >>> a = A() >>> hasattr(a, 'x') False >>> b = deco(a); assert b.x == 1; # b has an x and it's 1 >>> b = deco()(a); assert b.x == 1; # b has an x and it's 1 >>> b = deco(a, x=42); assert b.x == 42 # b has an x and it's 42 >>> b = deco(x=42)(a); assert b.x == 42; # b has an x and it's 42 ``` #### WARNING Note though that the type of `b` is not the same type as `a` ```pycon >>> isinstance(b, a.__class__) False ``` No, `b` is an instance of a `dol.base.Store`, which is a class containing an instance of a store (here, `a`). ```pycon >>> type(b) >>> b.store == a True ``` Now, here’s some more example, slightly closer to real usage ```pycon >>> from dol.trans import store_decorator >>> from inspect import signature >>> >>> def rm_deletion(store=None, *, msg='Deletions not allowed.'): ... name = getattr(store, '__name__', 'Something') + '_w_sommething' ... assert isinstance(store, type), f"Should be a type, was {type(store)}: {store}" ... wrapped_store = type(name, (store,), {}) ... wrapped_store.__delitem__ = lambda self, k: msg ... return wrapped_store ... >>> remove_deletion = store_decorator(rm_deletion) ``` See how the signature of the wrapper has some extra inputs that were injected (_\_module_\_, \_\_qualname_\_, etc.): ```pycon >>> print(str(signature(remove_deletion))) (store=None, *, msg='Deletions not allowed.', __module__=None, __name__=None, __qualname__=None, __doc__=None, __annotations__=None, __defaults__=None, __kwdefaults__=None) ``` Using it as a class decorator factory (the most common way): As a class decorator “factory”, without parameters (and without ()): ```pycon >>> from collections import UserDict >>> @remove_deletion ... class WD(UserDict): ... "Here's the doc" ... pass >>> wd = WD(x=5, y=7) >>> assert wd == UserDict(x=5, y=7) # same as far as dict comparison goes >>> assert wd.__delitem__('x') == 'Deletions not allowed.' >>> assert wd.__doc__ == "Here's the doc" ``` As a class decorator “factory”, with parameters: ```pycon >>> @remove_deletion(msg='No way. I do not trust you!!') ... class WD(UserDict): ... >>> wd = WD(x=5, y=7) >>> assert wd == UserDict(x=5, y=7) # same as far as dict comparison goes >>> assert wd.__delitem__('x') == 'No way. I do not trust you!!' ``` The \_\_doc_\_ is empty: ```pycon >>> assert WD.__doc__ == None ``` But we could specify a doc if we wanted to: ```pycon >>> @remove_deletion(__doc__="Hi, I'm a doc.") ... class WD(UserDict): ... "This is the original doc, that will be overritten" >>> assert WD.__doc__ == "Hi, I'm a doc." ``` The class decorations above are equivalent to the two following: ```pycon >>> WD = remove_deletion(UserDict) >>> wd = WD(x=5, y=7) >>> assert wd == UserDict(x=5, y=7) # same as far as dict comparison goes >>> assert wd.__delitem__('x') == 'Deletions not allowed.' >>> >>> WD = remove_deletion(UserDict, msg='No way. I do not trust you!!') >>> wd = WD(x=5, y=7) >>> assert wd == UserDict(x=5, y=7) # same as far as dict comparison goes >>> assert wd.__delitem__('x') == 'No way. I do not trust you!!' ``` But we can also decorate instances. In this case they will be wrapped in a Store class before being passed on to the actual decorator. ```pycon >>> d = UserDict(x=5, y=7) >>> wd = remove_deletion(d) >>> assert wd == d # same as far as dict comparison goes >>> assert wd.__delitem__('x') == 'Deletions not allowed.' >>> >>> d = UserDict(x=5, y=7) >>> wd = remove_deletion(d, msg='No way. I do not trust you!!') >>> assert wd == d # same as far as dict comparison goes >>> assert wd.__delitem__('x') == 'No way. I do not trust you!!' ``` ### dol.trans.store_wrap(obj) Wrap a class or an instance in a `Store` (a class gets a `Store` subclass whose `__init__` builds the wrapped instance). ### dol.trans.take_everything(key) Key filter that accepts every key. ### dol.trans.transparent_key_method(self, k) Return the key as is (the default `getitem` of `mk_kv_reader_from_kv_collection`). ### dol.trans.warn_and_ignore_if_error(store=None, \*, errors=(, ), warn_msg='Ignoring error in \_\_getitem_\_ for key {k}: {e}') Like `ignore_if_error`, but also emit a warning (`warn_msg`) for each ignored error. ### dol.trans.wrap_kvs(store=None, , wrapper=None, name=None, key_of_id=None, id_of_key=None, obj_of_data=None, data_of_obj=None, preset=None, postget=None, key_codec=None, value_codec=None, key_encoder=None, key_decoder=None, value_encoder=None, value_decoder=None, \_\_module_\_=None, outcoming_key_methods=(), outcoming_value_methods=(), ingoing_key_methods=(), ingoing_value_methods=(), \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None) Make a Store that is wrapped with the given key/val transformers. Naming convention: ```default Morphemes: key: outer key _id: inner key obj: outer value data: inner value Grammar: Y_of_X: means that you get a Y output when giving an X input. Also known as X_to_Y. ``` * **Parameters:** * **store** – Store class or instance * **name** – Name to give the wrapper class * **key_of_id** – The outcoming key transformation function. Forms are `k = key_of_id(_id)` or `k = key_of_id(self, _id)` * **id_of_key** – The ingoing key transformation function. Forms are `_id = id_of_key(k)` or `_id = id_of_key(self, k)` * **obj_of_data** – The outcoming val transformation function. Forms are `obj = obj_of_data(data)` or `obj = obj_of_data(self, data)` * **data_of_obj** – The ingoing val transformation function. Forms are `data = data_of_obj(obj)` or `data = data_of_obj(self, obj)` * **preset** – A function that is called before doing a `__setitem__`. The function is called with both `k` and `v` as inputs, and should output a transformed value. The intent use is to do ingoing value transformations conditioned on the key. For example, you may want to serialize an object depending on if you’re writing to a ‘.csv’, or ‘.json’, or ‘.pickle’ file. Forms are `preset(k, obj)` or `preset(self, k, obj)` * **postget** – A function that is called after the value `v` for a key `k` is be `__getitem__`. The function is called with both `k` and `v` as inputs, and should output a transformed value. The intent use is to do outcoming value transformations conditioned on the key. We already have `obj_of_data` for outcoming value trans, but cannot condition it’s behavior on k. For example, you may want to deserialize the bytes of a ‘.csv’, or ‘.json’, or ‘.pickle’ in different ways. Forms are `obj = postget(k, data)` or `obj = postget(self, k, data)` * **Returns:** A key and/or value transformed wrapped (or wrapper) class (or instance). ```pycon >>> def key_of_id(_id): ... return _id.upper() >>> def id_of_key(k): ... return k.lower() >>> def obj_of_data(data): ... return data - 100 >>> def data_of_obj(obj): ... return obj + 100 >>> >>> A = wrap_kvs(dict, name='A', ... key_of_id=key_of_id, id_of_key=id_of_key, obj_of_data=obj_of_data, data_of_obj=data_of_obj) >>> a = A() >>> a['KEY'] = 1 >>> a # repr is just the base class (dict) repr, so shows "inside" the store (lower case keys and +100) {'key': 101} >>> a['key'] = 2 >>> print(a) # repr is just the base class (dict) repr, so shows "inside" the store (lower case keys and +100) {'key': 102} >>> a['kEy'] = 3 >>> a # repr is just the base class (dict) repr, so shows "inside" the store (lower case keys and +100) {'key': 103} >>> list(a) # but from the point of view of the interface the keys are all upper case ['KEY'] >>> list(a.items()) # and the values are those we put there. [('KEY', 3)] >>> >>> # And now this: Showing how to condition the value transform (like obj_of_data), but conditioned on key. >>> B = wrap_kvs(dict, name='B', postget=lambda k, v: f'upper {v}' if k[0].isupper() else f'lower {v}') >>> b = B() >>> b['BIG'] = 'letters' >>> b['small'] = 'text' >>> list(b.items()) [('BIG', 'upper letters'), ('small', 'lower text')] >>> >>> >>> # Let's try preset and postget. We'll wrap a dict and write the same list of lists object to >>> # keys ending with .csv, .json, and .pkl, specifying the obvious extension-dependent >>> # serialization/deserialization we want to associate with it. >>> >>> # First, some very simple csv transformation functions >>> to_csv = lambda LoL: '\\n'.join(map(','.join, map(lambda L: (x for x in L), LoL))) >>> from_csv = lambda csv: list(map(lambda x: x.split(','), csv.split('\\n'))) >>> LoL = [['a','b','c'],['d','e','f']] >>> assert from_csv(to_csv(LoL)) == LoL >>> >>> import json, pickle >>> >>> def preset(k, v): ... if k.endswith('.csv'): ... return to_csv(v) ... elif k.endswith('.json'): ... return json.dumps(v) ... elif k.endswith('.pkl'): ... return pickle.dumps(v) ... else: ... return v # as is ... ... >>> def postget(k, v): ... if k.endswith('.csv'): ... return from_csv(v) ... elif k.endswith('.json'): ... return json.loads(v) ... elif k.endswith('.pkl'): ... return pickle.loads(v) ... else: ... return v # as is ... >>> mydict = wrap_kvs(dict, preset=preset, postget=postget) >>> >>> obj = [['a','b','c'],['d','e','f']] >>> d = mydict() >>> d['foo.csv'] = obj # store the object as csv >>> d # "printing" a dict by-passes the transformations, so we see the data in the "raw" format it is stored in. {'foo.csv': 'a,b,c\\nd,e,f'} >>> d['foo.csv'] # but if we actually ask for the data, it deserializes to our original object [['a', 'b', 'c'], ['d', 'e', 'f']] >>> d['bar.json'] = obj # store the object as json >>> d {'foo.csv': 'a,b,c\\nd,e,f', 'bar.json': '[["a", "b", "c"], ["d", "e", "f"]]'} >>> d['bar.json'] [['a', 'b', 'c'], ['d', 'e', 'f']] >>> d['bar.json'] = {'a': 1, 'b': [1, 2], 'c': 'normal json'} # let's write a normal json instead. >>> d {'foo.csv': 'a,b,c\\nd,e,f', 'bar.json': '{"a": 1, "b": [1, 2], "c": "normal json"}'} >>> del d['foo.csv'] >>> del d['bar.json'] >>> d['foo.pkl'] = obj # 'save' obj as pickle >>> d['foo.pkl'] [['a', 'b', 'c'], ['d', 'e', 'f']] ``` # _autosummary/dol.trash.html.md # dol.trash Cross-platform file trash/recycle bin functionality for dol. This module provides configurable file deletion strategies with support for moving files to trash/recycle bin instead of permanent deletion. Available deletion strategies: > - default_delete_func: Safe trash with warning on fallback to os.remove > - permanent_delete: Direct os.remove (no warnings) > - trash_only: Error if trash unavailable Configure deletion behavior when creating file stores by passing the `delete_func` parameter or setting the `_delete_func` class attribute. ### Functions | [`default_delete_func`](_autosummary/dol.trash.html.md#dol.trash.default_delete_func)(filepath) | Try trash, fall back to permanent delete on failure. | |----------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | [`get_platform_trash_func`](_autosummary/dol.trash.html.md#dol.trash.get_platform_trash_func)() | Get platform-specific trash function with caching. | | [`make_safe_delete_func`](_autosummary/dol.trash.html.md#dol.trash.make_safe_delete_func)([...]) | Create a deletion function that tries trash first, falls back to permanent delete. | | [`permanent_delete`](_autosummary/dol.trash.html.md#dol.trash.permanent_delete)(filepath) | Permanently delete a file (no trash, no warnings). | | [`trash_only`](_autosummary/dol.trash.html.md#dol.trash.trash_only)(filepath) | Move to trash only - raise error if trash unavailable. | ### dol.trash.default_delete_func(filepath) Try trash, fall back to permanent delete on failure. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### dol.trash.get_platform_trash_func() Get platform-specific trash function with caching. Returns None if no trash function is available. Priority order: 1. send2trash library (if installed) 2. Platform-specific implementation (macOS, Windows, Linux) 3. None (will fall back to os.remove) * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]] * **Returns:** Deletion function that moves files to trash, or None if unavailable ### dol.trash.make_safe_delete_func(permanent_delete_func=, warn_on_fallback=True) Create a deletion function that tries trash first, falls back to permanent delete. * **Parameters:** * **permanent_delete_func** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – Function to use if trash is unavailable * **warn_on_fallback** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to warn when falling back to permanent delete * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`None`](https://docs.python.org/3/builtins/constants.html#None)] * **Returns:** A deletion function that tries trash with fallback ### dol.trash.permanent_delete(filepath) Permanently delete a file (no trash, no warnings). * **Parameters:** **filepath** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Path to file to delete * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) Use this function when you want permanent deletion without warnings. Pass it as delete_func parameter: Files(‘/my/data’, delete_func=permanent_delete) ### dol.trash.trash_only(filepath) Move to trash only - raise error if trash unavailable. * **Parameters:** **filepath** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Path to file to trash * **Raises:** [**RuntimeError**](https://docs.python.org/3/builtins/exceptions.html#RuntimeError) – If trash functionality not available * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) Use this function when you want to ensure files are only moved to trash, never permanently deleted. Raises error if trash is unavailable. Pass it as delete_func parameter: Files(‘/my/data’, delete_func=trash_only) # _autosummary/dol.util.html.md # dol.util General util objects: function composition, grouping, partial classes, file helpers. Main entry points: - `Pipe`: compose functions left to right - `partialclass`: `functools.partial` for classes - `groupby`, `regroupby`, `igroupby`: group items by a key function - `chain_get`: first value found for a sequence of keys - `written_bytes`, `read_from_bytes`: turn file-writing/reading functions into bytes codecs ```pycon >>> from dol.util import Pipe >>> Pipe(lambda x: x + 1, str)(1) '2' ``` ### Functions | [`add_as_attribute_of`](_autosummary/dol.util.html.md#dol.util.add_as_attribute_of)(obj[, name]) | Decorator that adds a function as an attribute of a container object `obj`. | |-----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`add_attrs`](_autosummary/dol.util.html.md#dol.util.add_attrs)([remember_added_attrs, if_attr_exists]) | Make a function that will add attributes to an obj. | | `attrs_of`(obj) | | | [`chain_get`](_autosummary/dol.util.html.md#dol.util.chain_get)(d, keys[, default]) | Returns the `d[key]` value for the first `key` in `keys` that is in `d`, and default if none are found | | [`copy_attrs`](_autosummary/dol.util.html.md#dol.util.copy_attrs)(target, source, attrs[, ...]) | Copy attributes from one object to another. | | `copy_attrs_from`(from_obj, to_obj, attrs) | | | [`decorate_callables`](_autosummary/dol.util.html.md#dol.util.decorate_callables)(decorator[, cls]) | Decorate all (non-underscored) callables in a class with a decorator. | | `delegate_as`(delegate_cls[, to, include, exclude]) | | | [`fill_with_dflts`](_autosummary/dol.util.html.md#dol.util.fill_with_dflts)(d[, dflt_dict]) | Fed up with multiline handling of dict arguments? Fed up of repeating the if d is None: d = {} lines ad nauseam (because defaults can't be dicts as a default because dicts are mutable blah blah, and the python kings don't seem to think a mutable dict is useful enough)? Well, my favorite solution would be a built-in handling of the problem of complex/smart defaults, that is visible in the code and in the docs. | | [`flatten_pipe`](_autosummary/dol.util.html.md#dol.util.flatten_pipe)(pipe) | Unravel nested Pipes to get a flat 'sequence of functions' version of input. | | [`format_invocation`](_autosummary/dol.util.html.md#dol.util.format_invocation)([name, args, kwargs]) | Given a name, positional arguments, and keyword arguments, format a basic Python-style function call. | | `fullpath`(path) | | | `function_info_string`(func) | | | [`get_app_folder`](_autosummary/dol.util.html.md#dol.util.get_app_folder)([folder_kind]) | Get the full path of a directory suitable for storing application-specific configs, (or data, or cache, or state or runtime) | | [`groupby`](_autosummary/dol.util.html.md#dol.util.groupby)(items, key[, val, group_factory]) | Groups items according to group keys updated from those items through the given `key` function (mapping an item to its group key). | | [`has_enabled_clear_method`](_autosummary/dol.util.html.md#dol.util.has_enabled_clear_method)(store) | Returns True iff obj has a clear method that is enabled (i.e. not disabled). | | `identity_func`(x) | | | [`igroupby`](_autosummary/dol.util.html.md#dol.util.igroupby)(items, key[, val, group_factory, ...]) | The generator version of dol groupby. | | [`inject_method`](_autosummary/dol.util.html.md#dol.util.inject_method)(obj, method_function[, ...]) | method_function could be: | | [`instance_checker`](_autosummary/dol.util.html.md#dol.util.instance_checker)(\*types) | Makes a filter function that checks the type of an object. | | [`invertible_maps`](_autosummary/dol.util.html.md#dol.util.invertible_maps)([mapping, inv_mapping]) | Returns two maps that are inverse of each other. | | [`is_classmethod`](_autosummary/dol.util.html.md#dol.util.is_classmethod)(obj) | Checks if an object is a classmethod. | | [`is_unbound_method`](_autosummary/dol.util.html.md#dol.util.is_unbound_method)(obj) | Determines if the given object is an unbound method. | | [`max_common_prefix`](_autosummary/dol.util.html.md#dol.util.max_common_prefix)(a, \*[, default]) | Given a list of strings (or other sliceable seq), returns the longest common prefix | | `move_files_of_folder_to_trash`(folder) | | | [`named_partial`](_autosummary/dol.util.html.md#dol.util.named_partial)(func, \*args[, \_\_name_\_]) | functools.partial, but with a \_\_name_\_ | | `nest_in_dict`(keys, values) | | | [`non_colliding_key`](_autosummary/dol.util.html.md#dol.util.non_colliding_key)(key, exclude, \*[, ...]) | Return a key not present in the exclude container. | | [`norm_kv_filt`](_autosummary/dol.util.html.md#dol.util.norm_kv_filt)(kv_filt) | Prepare a boolean function to be used with `filter` when fed an iterable of (k, v) pairs. | | [`not_a_mac_junk_path`](_autosummary/dol.util.html.md#dol.util.not_a_mac_junk_path)(path) | A function that will tell you if the path is not a mac junk path/ More precisely, doesn't end with '.DS_Store' or have a `__MACOSX` folder somewhere on it's way. | | `ntup`(\*\*kwargs) | | | [`num_of_args`](_autosummary/dol.util.html.md#dol.util.num_of_args)(func) | Number of arguments (parameters) of the function. | | [`num_of_required_args`](_autosummary/dol.util.html.md#dol.util.num_of_required_args)(func) | Number or REQUIRED arguments of a function. | | [`partialclass`](_autosummary/dol.util.html.md#dol.util.partialclass)(cls, \*args, \*\*kwargs) | What `partial(cls, *args, **kwargs)` does, but returning a class instead of an object. | | [`read_from_bytes`](_autosummary/dol.util.html.md#dol.util.read_from_bytes)(file_reader[, obj, ...]) | Takes a file reading function that expects a file-like object, and returns a function that instead of reading from a file, reads from bytes. | | [`regroupby`](_autosummary/dol.util.html.md#dol.util.regroupby)(items, \*key_funcs, \*\*named_key_funcs) | Recursive groupby. | | [`safe_compile`](_autosummary/dol.util.html.md#dol.util.safe_compile)(path[, normalize_path]) | Compile a *literal file path* into a regex pattern that matches that path, normalizing separators and escaping regex-special characters on Windows. | | `signature_string_or_default`(func[, default]) | | | `single_nest_in_dict`(key, value) | | | `static_identity_method`(x) | | | [`str_to_var_str`](_autosummary/dol.util.html.md#dol.util.str_to_var_str)(s) | Make a valid python variable string from the input string. | | [`truncate_string_with_marker`](_autosummary/dol.util.html.md#dol.util.truncate_string_with_marker)(s, \*[, ...]) | Return a string with a limited length. | | `write_to_file`(obj, key) | | | [`written_bytes`](_autosummary/dol.util.html.md#dol.util.written_bytes)(file_writer[, obj, ...]) | Takes a file writing function that expects an object and a file-like object, and returns a function that instead of writing to a file, returns the bytes that would have been written. | | [`written_key`](_autosummary/dol.util.html.md#dol.util.written_key)([obj, writer, key, ...]) | Writes an object to a key and returns the key. | ### Classes | [`AttributeMapping`](_autosummary/dol.util.html.md#dol.util.AttributeMapping) | A read-only mapping with attribute access. | |------------------------------------------------------------------------------------|-----------------------------------------------------------------------------| | [`AttributeMutableMapping`](_autosummary/dol.util.html.md#dol.util.AttributeMutableMapping) | A mutable mapping that provides both attribute and dictionary-style access. | | `DelegatedAttribute`(delegate_name, attr_name) | | | [`FolderSpec`](_autosummary/dol.util.html.md#dol.util.FolderSpec)(env_var, default_path) | | | `HashableMixin`() | | | `ImmutableMixin`() | | | [`LiteralVal`](_autosummary/dol.util.html.md#dol.util.LiteralVal)(val) | An object to indicate that the value should be considered literally. | | `ModuleNotFoundErrorNiceMessage`([msg]) | | | `ModuleNotFoundIgnore`() | | | `ModuleNotFoundWarning`([msg]) | | | `MutableStruct`(\*\*attr_val_dict) | | | [`Pipe`](_autosummary/dol.util.html.md#dol.util.Pipe)(\*funcs, \*\*named_funcs) | Simple function composition. | | `SimpleProperty`() | | | `Struct`(\*\*attr_val_dict) | | | [`imdict`](_autosummary/dol.util.html.md#dol.util.imdict) | A frozen hashable dict | | [`lazyprop`](_autosummary/dol.util.html.md#dol.util.lazyprop)(func) | A descriptor implementation of lazyprop (cached property). | | [`lazyprop_w_sentinel`](_autosummary/dol.util.html.md#dol.util.lazyprop_w_sentinel)(func) | A descriptor implementation of lazyprop (cached property). | | [`staticproperty`](_autosummary/dol.util.html.md#dol.util.staticproperty)(function) | A decorator for defining static properties in classes. | ### *class* dol.util.AttributeMapping Bases: [`SimpleNamespace`](https://docs.python.org/3/library/types.html#types.SimpleNamespace), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] A read-only mapping with attribute access. Useful when you want mapping interface but don’t need mutation. ### Examples ```pycon >>> ns = AttributeMapping(x=10, y=20) >>> ns.x 10 >>> ns['y'] 20 >>> list(ns) ['x', 'y'] ``` #### *classmethod* from_mapping(mapping) Create an AttributeMapping from a regular mapping. This is useful when you want to convert a dictionary or other mapping into an AttributeMapping for attribute-style access. * **Return type:** [`AttributeMapping`](_autosummary/dol.util.html.md#dol.util.AttributeMapping) ### *class* dol.util.AttributeMutableMapping Bases: [`AttributeMapping`](_autosummary/dol.util.html.md#dol.util.AttributeMapping), [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] A mutable mapping that provides both attribute and dictionary-style access. Extends AttributeMapping with mutation capabilities, ensuring proper error handling and protocol compliance. ### Examples ```pycon >>> ns = AttributeMutableMapping(apple=1, banana=2) >>> ns.apple 1 >>> ns['banana'] 2 >>> ns['cherry'] = 3 >>> ns.cherry 3 >>> list(ns) ['apple', 'banana', 'cherry'] >>> len(ns) 3 >>> 'apple' in ns True >>> del ns['banana'] >>> 'banana' in ns False ``` ### *class* dol.util.FolderSpec(env_var, default_path) Bases: [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) #### default_path Alias for field number 1 #### env_var Alias for field number 0 ### *class* dol.util.LiteralVal(val) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An object to indicate that the value should be considered literally. ```pycon >>> t = LiteralVal(42) >>> t.get_val() 42 >>> t() 42 ``` #### get_val() Get the value wrapped by LiteralVal instance. One might want to use `literal.get_val()` instead `literal()` to get the value a `LiteralVal` is wrapping because `.get_val` is more explicit. That said, with a bit of hesitation, we allow the `literal()` form as well since it is useful in situations where we need to use a callback function to get a value. ### *class* dol.util.Pipe(\*funcs, \*\*named_funcs) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Simple function composition. That is, gives you a callable that implements input -> f_1 -> … -> f_n -> output. ```pycon >>> def foo(a, b=2): ... return a + b >>> f = Pipe(foo, lambda x: print(f"x: {x}")) >>> f(3) x: 5 >>> len(f) 2 ``` You can name functions, but this would just be for documentation purposes. The names are completely ignored. ```pycon >>> g = Pipe( ... add_numbers = lambda x, y: x + y, ... multiply_by_2 = lambda x: x * 2, ... stringify = str ... ) >>> g(2, 3) '10' >>> len(g) 3 ``` ### Notes - Pipe instances don’t have a \_\_name_\_ etc. So some expectations of normal functions are not met. - Pipe instance are pickalable (as long as the functions that compose them are) You can specify a single functions: ```pycon >>> Pipe(lambda x: x + 1)(2) 3 ``` but ```pycon >>> Pipe() Traceback (most recent call last): ... ValueError: You need to specify at least one function! ``` You can specify an instance name and/or doc with the special (reserved) argument names `__name__` and `__doc__` (which therefore can’t be used as function names): ```pycon >>> f = Pipe(map, add_it=sum, __name__='map_and_sum', __doc__='Apply func and add') >>> f(lambda x: x * 10, [1, 2, 3]) 60 >>> f.__name__ 'map_and_sum' >>> f.__doc__ 'Apply func and add' ``` ### dol.util.add_as_attribute_of(obj, name=None) Decorator that adds a function as an attribute of a container object `obj`. If no `name` is given, the `__name__` of the function will be used, with a leading underscore removed. This is useful for adding helper functions to main “container” functions without polluting the namespace of the module, at least from the point of view of imports and tab completion. ```pycon >>> def foo(): ... pass >>> >>> @add_as_attribute_of(foo) ... def _helper(): ... pass >>> hasattr(foo, 'helper') True >>> callable(foo.helper) True ``` In reality, any object that has a `__name__` can be added to the attribute of `obj`, but the intention is to add helper functions to main “container” functions. ### dol.util.add_attrs(remember_added_attrs=True, if_attr_exists='raise', \*\*attrs) Make a function that will add attributes to an obj. Originally meant to be used as a decorator of a function, to inject ```pycon >>> from dol.util import add_attrs >>> @add_attrs(bar='bituate', hello='world') ... def foo(): ... pass >>> [x for x in dir(foo) if not x.startswith('_')] ['bar', 'hello'] >>> foo.bar 'bituate' >>> foo.hello 'world' >>> foo._added_attrs # Another attr was added to hold the list of attributes added (in case we need to remove them ['bar', 'hello'] ``` ### dol.util.chain_get(d, keys, default=None) Returns the `d[key]` value for the first `key` in `keys` that is in `d`, and default if none are found #### NOTE Think of `collections.ChainMap` where you can look for a single key in a sequence of maps until we find it. Here we look for a sequence of keys in a single map, stopping as soon as we find a key that the map has. ```pycon >>> d = {'here': '&', 'there': 'and', 'every': 'where'} >>> chain_get(d, ['not there', 'not there either', 'there', 'every']) 'and' ``` Notice how `'not there'` and `'not there either'` are skipped, `'there'` is found and used to retrieve the value, and `'every'` is not even checked (because `'there'` was found). If non of the keys are found, `None` is returned by default. ```pycon >>> assert chain_get(d, ('none', 'of', 'these')) is None ``` You can change this default though: ```pycon >>> chain_get(d, ('none', 'of', 'these'), default='Not Found') 'Not Found' ``` ### dol.util.copy_attrs(target, source, attrs, raise_error_if_an_attr_is_missing=True) Copy attributes from one object to another. ```pycon >>> class A: ... x = 0 >>> class B: ... x = 1 ... yy = 2 ... zzz = 3 >>> dict_of = lambda o: {a: getattr(o, a) for a in dir(A) if not a.startswith('_')} >>> dict_of(A) {'x': 0} >>> copy_attrs(A, B, 'yy') >>> dict_of(A) {'x': 0, 'yy': 2} >>> copy_attrs(A, B, ['x', 'zzz']) >>> dict_of(A) {'x': 1, 'yy': 2, 'zzz': 3} ``` But if you try to copy something that `B` (the source) doesn’t have, copy_attrs will complain: ```pycon >>> copy_attrs(A, B, 'this_is_not_an_attr') Traceback (most recent call last): ... AttributeError: type object 'B' has no attribute 'this_is_not_an_attr' ``` If you tell it not to complain, it’ll just ignore attributes that are not in source. ```pycon >>> copy_attrs(A, B, ['nothing', 'here', 'exists'], raise_error_if_an_attr_is_missing=False) >>> dict_of(A) {'x': 1, 'yy': 2, 'zzz': 3} ``` ### dol.util.decorate_callables(decorator, cls=None) Decorate all (non-underscored) callables in a class with a decorator. ```pycon >>> from dol.util import LiteralVal >>> @decorate_callables(property) ... class A: ... def wet(self): ... return 'dry' ... @LiteralVal ... def big(self): ... return 'small' >>> a = A() >>> a.wet 'dry' >>> a.big() 'small' ``` ### dol.util.fill_with_dflts(d, dflt_dict=None) Fed up with multiline handling of dict arguments? Fed up of repeating the if d is None: d = {} lines ad nauseam (because defaults can’t be dicts as a default because dicts are mutable blah blah, and the python kings don’t seem to think a mutable dict is useful enough)? Well, my favorite solution would be a built-in handling of the problem of complex/smart defaults, that is visible in the code and in the docs. But for now, here’s one of the tricks I use. Main use is to handle defaults of function arguments. Say you have a function `func(d=None)` and you want `d` to be a dict that has at least the keys `foo` and `bar` with default values 7 and 42 respectively. Then, in the beginning of your function code you’ll say: > d = fill_with_dflts(d, {‘a’: 7, ‘b’: 42}) See examples to know how to use it. #### ATTENTION A shallow copy of the dict is made. Know how that affects you (or not). #### ATTENTION This is not recursive: It won’t be filling any nested fields with defaults. * **Parameters:** * **d** – The dict you want to “fill” * **dflt_dict** – What to fill it with (a {k: v, …} dict where if k is missing in d, you’ll get a new field k, with value v. * **Returns:** val entries (if the key was missing in d). * **Return type:** a dict with the new key ```pycon >>> fill_with_dflts(None) {} >>> fill_with_dflts(None, {'a': 7, 'b': 42}) {'a': 7, 'b': 42} >>> fill_with_dflts({}, {'a': 7, 'b': 42}) {'a': 7, 'b': 42} >>> fill_with_dflts({'b': 1000}, {'a': 7, 'b': 42}) {'a': 7, 'b': 1000} ``` ### dol.util.flatten_pipe(pipe) Unravel nested Pipes to get a flat ‘sequence of functions’ version of input. ```pycon >>> def f(x): return x + 1 >>> def g(x): return x * 2 >>> def h(x): return x - 3 >>> a = Pipe(f, g, h) >>> b = Pipe(f, Pipe(g, h)) >>> len(a) 3 >>> len(b) 2 >>> c = flatten_pipe(b) >>> len(c) 3 >>> assert a(10) == b(10) == c(10) == 19 ``` ### dol.util.format_invocation(name='', args=(), kwargs=None) Given a name, positional arguments, and keyword arguments, format a basic Python-style function call. ```pycon >>> print(format_invocation('func', args=(1, 2), kwargs={'c': 3})) func(1, 2, c=3) >>> print(format_invocation('a_func', args=(1,))) a_func(1) >>> print(format_invocation('kw_func', kwargs=[('a', 1), ('b', 2)])) kw_func(a=1, b=2) ``` ### dol.util.get_app_config_folder(, folder_kind='config') Get the full path of a directory suitable for storing application-specific configs, (or data, or cache, or state or runtime) On Windows, this is typically %APPDATA%. On macOS, this is typically ~/.config. On Linux, this is typically ~/.config. * **Parameters:** **folder_kind** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The kind of folder to get. One of ‘config’, ‘data’, ‘cache’, ‘state’, ‘runtime’. Defaults to ‘config’. Here are concise explanations for each folder kind: **config**: User preferences and settings files (e.g., API keys, theme preferences, editor settings). Files users might edit manually or that define how the app behaves. **data**: Essential user-created content and application state (e.g., databases, saved games, user documents, session files). Data that should be backed up and persists across updates. **cache**: Temporary, regeneratable files (e.g., downloaded images, compiled assets, web cache). Can be safely deleted to free space without losing user work. **state**: Application state and logs that persist between sessions but aren’t critical user data (e.g., command history, undo history, recently opened files, log files). Unlike cache, shouldn’t be auto-deleted. **runtime**: Temporary runtime files that only exist while the app runs (e.g., PID files, Unix sockets, lock files, named pipes). Typically cleared on logout/reboot. **TL;DR**: config = settings, data = user files, cache = disposable, state = logs/history, runtime = process files. * **Returns:** The full path of the app data folder. * **Return type:** [*str*](https://docs.python.org/3/builtins/stdtypes.html#str) See [https://github.com/i2mint/i2mint/issues/1](https://github.com/i2mint/i2mint/issues/1). ### dol.util.get_app_data_folder(, folder_kind='data') Get the full path of a directory suitable for storing application-specific configs, (or data, or cache, or state or runtime) On Windows, this is typically %APPDATA%. On macOS, this is typically ~/.config. On Linux, this is typically ~/.config. * **Parameters:** **folder_kind** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The kind of folder to get. One of ‘config’, ‘data’, ‘cache’, ‘state’, ‘runtime’. Defaults to ‘config’. Here are concise explanations for each folder kind: **config**: User preferences and settings files (e.g., API keys, theme preferences, editor settings). Files users might edit manually or that define how the app behaves. **data**: Essential user-created content and application state (e.g., databases, saved games, user documents, session files). Data that should be backed up and persists across updates. **cache**: Temporary, regeneratable files (e.g., downloaded images, compiled assets, web cache). Can be safely deleted to free space without losing user work. **state**: Application state and logs that persist between sessions but aren’t critical user data (e.g., command history, undo history, recently opened files, log files). Unlike cache, shouldn’t be auto-deleted. **runtime**: Temporary runtime files that only exist while the app runs (e.g., PID files, Unix sockets, lock files, named pipes). Typically cleared on logout/reboot. **TL;DR**: config = settings, data = user files, cache = disposable, state = logs/history, runtime = process files. * **Returns:** The full path of the app data folder. * **Return type:** [*str*](https://docs.python.org/3/builtins/stdtypes.html#str) See [https://github.com/i2mint/i2mint/issues/1](https://github.com/i2mint/i2mint/issues/1). ### dol.util.get_app_folder(folder_kind='config') Get the full path of a directory suitable for storing application-specific configs, (or data, or cache, or state or runtime) On Windows, this is typically %APPDATA%. On macOS, this is typically ~/.config. On Linux, this is typically ~/.config. * **Parameters:** **folder_kind** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'config'`, `'data'`, `'cache'`, `'state'`, `'runtime'`]) – The kind of folder to get. One of ‘config’, ‘data’, ‘cache’, ‘state’, ‘runtime’. Defaults to ‘config’. Here are concise explanations for each folder kind: **config**: User preferences and settings files (e.g., API keys, theme preferences, editor settings). Files users might edit manually or that define how the app behaves. **data**: Essential user-created content and application state (e.g., databases, saved games, user documents, session files). Data that should be backed up and persists across updates. **cache**: Temporary, regeneratable files (e.g., downloaded images, compiled assets, web cache). Can be safely deleted to free space without losing user work. **state**: Application state and logs that persist between sessions but aren’t critical user data (e.g., command history, undo history, recently opened files, log files). Unlike cache, shouldn’t be auto-deleted. **runtime**: Temporary runtime files that only exist while the app runs (e.g., PID files, Unix sockets, lock files, named pipes). Typically cleared on logout/reboot. **TL;DR**: config = settings, data = user files, cache = disposable, state = logs/history, runtime = process files. * **Returns:** The full path of the app data folder. * **Return type:** [*str*](https://docs.python.org/3/builtins/stdtypes.html#str) See [https://github.com/i2mint/i2mint/issues/1](https://github.com/i2mint/i2mint/issues/1). ### dol.util.groupby(items, key, val=None, group_factory=) Groups items according to group keys updated from those items through the given `key` function (mapping an item to its group key). * **Parameters:** * **items** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – iterable of items * **key** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Hashable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable)]) – The function that computes a key from an item. Needs to return a hashable. * **val** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – An optional function that computes a val from an item. If not given, the item itself will be taken. * **group_factory** – The function to make new (empty) group objects and accumulate group items. group_items = group_factory() will be called to make a new empty group collection group_items.append(x) will be called to add x to that collection The default is `list` * **Returns:** items_in_that_group, …} * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### SEE ALSO regroupby, itertools.groupby, and dol.source.SequenceKvReader ```pycon >>> groupby(range(11), key=lambda x: x % 3) {0: [0, 3, 6, 9], 1: [1, 4, 7, 10], 2: [2, 5, 8]} >>> >>> tokens = ['the', 'fox', 'is', 'in', 'a', 'box'] >>> groupby(tokens, len) {3: ['the', 'fox', 'box'], 2: ['is', 'in'], 1: ['a']} >>> key_map = {1: 'one', 2: 'two'} >>> groupby(tokens, lambda x: key_map.get(len(x), 'more')) {'more': ['the', 'fox', 'box'], 'two': ['is', 'in'], 'one': ['a']} >>> stopwords = {'the', 'in', 'a', 'on'} >>> groupby(tokens, lambda w: w in stopwords) {True: ['the', 'in', 'a'], False: ['fox', 'is', 'box']} >>> groupby(tokens, lambda w: ['words', 'stopwords'][int(w in stopwords)]) {'stopwords': ['the', 'in', 'a'], 'words': ['fox', 'is', 'box']} ``` ### dol.util.has_enabled_clear_method(store) Returns True iff obj has a clear method that is enabled (i.e. not disabled) ### dol.util.igroupby(items, key, val=None, group_factory=, group_release_cond=>, release_remainding=True, append_to_group_items=, grouper_mapping=) The generator version of dol groupby. Groups items according to group keys updated from those items through the given `key` function (mapping an item to its group key), yielding the groups according to a logic defined by `group_release_cond` * **Parameters:** * **items** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – iterable of items * **key** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Hashable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable)]) – The function that computes a key from an item. Needs to return a hashable. * **val** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – An optional function that computes a val from an item. If not given, the item itself will be taken. * **group_factory** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[], [`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – The function to make new (empty) group objects and accumulate group items. group_items = group_collector() will be called to make a new empty group collection group_items.append(x) will be called to add x to that collection The default is `list` * **group_release_cond** (`Union`[[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Hashable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable), [`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict), [`Hashable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable), [`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]]) – A boolean function that will be applied, at every iteration, to the accumulated items of the group that was just updated, and determines (if True) if the (group_key, group_items) should be yielded. The default is False, which results in `lambda group_key, group_items: False` being used. * **release_remainding** – Once the input items have been consumed, there may still be some items in the grouping “cache”. `release_remainding` is a boolean that indicates whether the contents of this cache should be released or not. * **Yields:** `(group_key, items_in_that_group)` pairs The following will group numbers according to their parity (0 for even, 1 for odd), releasing a list of numbers collected when that list reaches length 3: ```pycon >>> g = igroupby(items=range(11), ... key=lambda x: x % 2, ... group_release_cond=lambda k, v: len(v) == 3) >>> list(g) [(0, [0, 2, 4]), (1, [1, 3, 5]), (0, [6, 8, 10]), (1, [7, 9])] ``` If we specify `release_remainding=False` though, we won’t get ```pycon >>> g = igroupby(items=range(11), ... key=lambda x: x % 2, ... group_release_cond=lambda k, v: len(v) == 3, ... release_remainding=False) >>> list(g) [(0, [0, 2, 4]), (1, [1, 3, 5]), (0, [6, 8, 10])] ``` # >>> grps = partial(igroupby, group_release_cond=False, release_remainding=True) Below we show that, with the default `group_release_cond = lambda k, v: False` and release_remainding=True\`\` we have `dict(igroupby(...)) == groupby(...)` ```pycon >>> from functools import partial >>> from dol import groupby >>> >>> kws = dict(items=range(11), key=lambda x: x % 3) >>> assert (dict(igroupby(**kws)) == groupby(**kws) ... == {0: [0, 3, 6, 9], 1: [1, 4, 7, 10], 2: [2, 5, 8]}) >>> >>> tokens = ['the', 'fox', 'is', 'in', 'a', 'box'] >>> kws = dict(items=tokens, key=len) >>> assert (dict(igroupby(**kws)) == groupby(**kws) ... == {3: ['the', 'fox', 'box'], 2: ['is', 'in'], 1: ['a']}) >>> >>> key_map = {1: 'one', 2: 'two'} >>> kws.update(key=lambda x: key_map.get(len(x), 'more')) >>> assert (dict(igroupby(**kws)) == groupby(**kws) ... == {'more': ['the', 'fox', 'box'], 'two': ['is', 'in'], 'one': ['a']}) >>> >>> stopwords = {'the', 'in', 'a', 'on'} >>> kws.update(key=lambda w: w in stopwords) >>> assert (dict(igroupby(**kws)) == groupby(**kws) ... == {True: ['the', 'in', 'a'], False: ['fox', 'is', 'box']}) >>> kws.update(key=lambda w: ['words', 'stopwords'][int(w in stopwords)]) >>> assert (dict(igroupby(**kws)) == groupby(**kws) ... == {'stopwords': ['the', 'in', 'a'], 'words': ['fox', 'is', 'box']}) ``` ### *class* dol.util.imdict Bases: `ImmutableMixin`, [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict), `HashableMixin` A frozen hashable dict ### dol.util.inject_method(obj, method_function, method_name=None) method_function could be: > * a function > * a {method_name: function, …} dict (for multiple injections) > * a list of functions or (function, method_name) pairs ### dol.util.instance_checker(\*types) Makes a filter function that checks the type of an object. ```pycon >>> f = instance_checker(int, float) >>> f(1) True >>> f(1.0) True >>> f('1.0') False ``` ### dol.util.invertible_maps(mapping=None, inv_mapping=None) Returns two maps that are inverse of each other. Raises an AssertionError iif both maps are None, or if the maps are not inverse of each other. Get a pair of invertible maps * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)] ```pycon >>> invertible_maps({1: 11, 2: 22}) ({1: 11, 2: 22}, {11: 1, 22: 2}) >>> invertible_maps(None, {11: 1, 22: 2}) ({1: 11, 2: 22}, {11: 1, 22: 2}) ``` You can specify one argument as an iterable (of keys for the mapping) and the other as a function (to be applied to the keys to get the inverse mapping). The function acts similarly to a `Mapping.__getitem__`, transforming each key to its associated value. The iterable defines the keys for the mapping, while the function is applied to each key to produce the values. ```pycon >>> invertible_maps([1,2,3], lambda x: x * 10) ({10: 1, 20: 2, 30: 3}, {1: 10, 2: 20, 3: 30}) >>> invertible_maps(lambda x: x * 10, [1,2,3]) ({1: 10, 2: 20, 3: 30}, {10: 1, 20: 2, 30: 3}) ``` If two maps are given and invertible, you just get them back ```pycon >>> invertible_maps({1: 11, 2: 22}, {11: 1, 22: 2}) ({1: 11, 2: 22}, {11: 1, 22: 2}) ``` Or if they’re not invertible ```pycon >>> invertible_maps({1: 11, 2: 22}, {11: 1, 22: 'ha, not what you expected!'}) Traceback (most recent call last): ... AssertionError: mapping and inv_mapping are not inverse of each other! ``` ```pycon >>> invertible_maps(None, None) Traceback (most recent call last): ... ValueError: You need to specify one or both maps ``` ### dol.util.is_classmethod(obj) Checks if an object is a classmethod. * **Parameters:** **obj** – The object to check. * **Returns:** True if the object is a classmethod, False otherwise. Example usage: ```pycon >>> class MyClass: ... @classmethod ... def class_method(cls): ... pass ... ... def instance_method(self): ... pass >>> obj1 = MyClass.class_method >>> obj2 = MyClass().instance_method >>> is_classmethod(obj1) True >>> is_classmethod(obj2) False ``` ### dol.util.is_unbound_method(obj) Determines if the given object is an unbound method. * **Parameters:** **obj** – The object to check. * **Returns:** True if obj is an unbound method, False otherwise. ### Examples ```pycon >>> import sys >>> import types >>> def function(): ... pass >>> class MyClass: ... def method(self): ... pass >>> is_unbound_method(MyClass.method) True >>> is_unbound_method(MyClass().method) False >>> is_unbound_method(function) False ``` ### *class* dol.util.lazyprop(func) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A descriptor implementation of lazyprop (cached property). Made based on David Beazley’s “Python Cookbook” book and enhanced with boltons.cacheutils ideas. ```pycon >>> class Test: ... def __init__(self, a): ... self.a = a ... @lazyprop ... def len(self): ... print('generating "len"') ... return len(self.a) >>> t = Test([0, 1, 2, 3, 4]) >>> t.__dict__ {'a': [0, 1, 2, 3, 4]} >>> t.len generating "len" 5 >>> t.__dict__ {'a': [0, 1, 2, 3, 4], 'len': 5} >>> t.len 5 >>> # But careful when using lazyprop that no one will change the value of a without deleting the property first >>> t.a = [0, 1, 2] # if we change a... >>> t.len # ... we still get the old cached value of len 5 >>> del t.len # if we delete the len prop >>> t.len # ... then len being recomputed again generating "len" 3 ``` ### *class* dol.util.lazyprop_w_sentinel(func) Bases: [`lazyprop`](_autosummary/dol.util.html.md#dol.util.lazyprop) A descriptor implementation of lazyprop (cached property). Inserts a `self.func.__name__ + '__cache_active'` attribute ```pycon >>> class Test: ... def __init__(self, a): ... self.a = a ... @lazyprop_w_sentinel ... def len(self): ... print('generating "len"') ... return len(self.a) >>> t = Test([0, 1, 2, 3, 4]) >>> lazyprop_w_sentinel.cache_is_active(t, 'len') False >>> t.__dict__ # let's look under the hood {'a': [0, 1, 2, 3, 4]} >>> t.len generating "len" 5 >>> lazyprop_w_sentinel.cache_is_active(t, 'len') True >>> t.len # notice there's no 'generating "len"' print this time! 5 >>> t.__dict__ # let's look under the hood {'a': [0, 1, 2, 3, 4], 'len': 5, 'sentinel_of__len': True} >>> # But careful when using lazyprop that no one will change the value of a without deleting the property first >>> t.a = [0, 1, 2] # if we change a... >>> t.len # ... we still get the old cached value of len 5 >>> del t.len # if we delete the len prop >>> t.len # ... then len being recomputed again generating "len" 3 ``` ### dol.util.max_common_prefix(a, , default='') Given a list of strings (or other sliceable seq), returns the longest common prefix * **Parameters:** **a** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)) – list-like of strings * **Returns:** the smallest common prefix of all strings in a ```pycon >>> max_common_prefix(['absolutely', 'abc', 'abba']) 'ab' >>> max_common_prefix(['absolutely', 'not', 'abc', 'abba']) '' >>> max_common_prefix([[3,2,1], [3,2,0]]) [3, 2] >>> max_common_prefix([[3,2,1], [3,2,0], [1,2,3]]) [] ``` If the input is empty, will return default (which defaults to ‘’). ```pycon >>> max_common_prefix([]) '' ``` If you want a different default, you can specify it with the default keyword argument. ```pycon >>> from functools import partial >>> my_max_common_prefix = partial(max_common_prefix, default=[]) >>> my_max_common_prefix([]) [] ``` ### dol.util.named_partial(func, \*args, \_\_name_\_=None, \*\*keywords) functools.partial, but with a \_\_name_\_ ```pycon >>> f = named_partial(print, sep='\n') >>> f.__name__ 'print' ``` ```pycon >>> f = named_partial(print, sep='\n', __name__='now_partial_has_a_name') >>> f.__name__ 'now_partial_has_a_name' ``` ### dol.util.non_colliding_key(key, exclude, , collision_handler=None, max_attempts=10000) Return a key not present in the exclude container. If the input key is already unique, it’s returned as-is. Otherwise, applies a collision_handler until a unique key is found. * **Parameters:** * **key** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)) – The candidate key to check/modify * **exclude** ([`Container`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Container)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)]) – Container of keys to avoid * **collision_handler** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`int`](https://docs.python.org/3/builtins/functions.html#int)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)]) – Function taking (key, attempt_number) and returning a modified key. For strings, defaults to appending “ (N)” suffix before extension. For other types, must be provided. * **max_attempts** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Maximum number of transformation attempts * **Return type:** [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`) * **Returns:** A key not present in the exclude container * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If no unique key found within max_attempts, or if collision_handler is None for non-string keys ```pycon >>> non_colliding_key("file.txt", set()) 'file.txt' >>> non_colliding_key("file.txt", {"file.txt"}) 'file (1).txt' >>> non_colliding_key("file.txt", {"file.txt", "file (1).txt"}) 'file (2).txt' >>> non_colliding_key(42, {42}, collision_handler=lambda k, n: k + n) 43 ``` ### dol.util.norm_kv_filt(kv_filt) Prepare a boolean function to be used with `filter` when fed an iterable of (k, v) pairs. So you have a mapping. Say a dict `d`. Now you want to go through d.items(), filtering based on the keys, or the values, or both. It’s not hard to do, really. If you’re using a dict you might use a dict comprehension, or in the general case you might do a `filter(lambda kv: my_filt(kv[0], kv[1]), d.items())` if you have a `my_filt` that works wiith k and v, etc. But thought simple, it can become a bit muddled. `norm_kv_filt` simplifies this by allowing you to bring your own filtering boolean function, whether it’s a key-based, value-based, or key-value-based one, and it will make a ready-to-use with `filter` function for you. Only thing: Your function needs to call a key `k` and a value `v`. But hey, it’s alright, if you have a function that calls things differently, just do something like ```python new_filt_func = lambda k, v: your_filt_func(..., key=k, ..., value=v, ...) ``` and all will be fine. * **Parameters:** **kv_filt** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – callable (starting with signature (k), (v), or (k, v)), and returning a boolean * **Returns:** A normalized callable. ```pycon >>> d = {'a': 1, 'b': 2, 'c': 3, 'd': 4} >>> list(filter(norm_kv_filt(lambda k: k in {'b', 'd'}), d.items())) [('b', 2), ('d', 4)] >>> list(filter(norm_kv_filt(lambda v: v > 2), d.items())) [('c', 3), ('d', 4)] >>> list(filter(norm_kv_filt(lambda k, v: (v > 1) & (k != 'c')), d.items())) [('b', 2), ('d', 4)] ``` ### dol.util.not_a_mac_junk_path(path) A function that will tell you if the path is not a mac junk path/ More precisely, doesn’t end with ‘.DS_Store’ or have a `__MACOSX` folder somewhere on it’s way. This is usually meant to be used with `filter` or `filt_iter` to “filter in” only those actually wanted files (not the junk that mac writes to your filesystem). These files annoyingly show up often in zip files, and are usually unwanted. See [https://apple.stackexchange.com/questions/239578/compress-without-ds-store-and-macosx](https://apple.stackexchange.com/questions/239578/compress-without-ds-store-and-macosx) ```pycon >>> paths = ['A/normal/path', 'A/__MACOSX/path', 'path/ending/in/.DS_Store', 'foo/b'] >>> list(filter(not_a_mac_junk_path, paths)) ['A/normal/path', 'foo/b'] ``` ### dol.util.num_of_args(func) Number of arguments (parameters) of the function. Contrast the behavior below with that of `num_of_required_args`. ```pycon >>> num_of_args(lambda a, b, c: None) 3 >>> num_of_args(lambda a, b, c=3: None) 3 >>> num_of_args(lambda a, *args, b, c=1, d=2, **kwargs: None) 6 ``` ### dol.util.num_of_required_args(func) Number or REQUIRED arguments of a function. Contrast the behavior below with that of `num_of_args`, which counts all parameters, including the variadics and defaulted ones. ```pycon >>> num_of_required_args(lambda a, b, c: None) 3 >>> num_of_required_args(lambda a, b, c=3: None) 2 >>> num_of_required_args(lambda a, *args, b, c=1, d=2, **kwargs: None) 2 ``` ### dol.util.partialclass(cls, \*args, \*\*kwargs) What `partial(cls, *args, **kwargs)` does, but returning a class instead of an object. * **Parameters:** * **cls** – Class to get the partial of * **kwargs** – The kwargs to fix The raison d’être of partialclass is that it returns a type, so let’s have a look at that with a useless class. ```pycon >>> from inspect import signature >>> class A: ... pass >>> assert isinstance(A, type) == isinstance(partialclass(A), type) == True ``` ```pycon >>> class A: ... def __init__(self, a=0, b=1): ... self.a, self.b = a, b ... def mysum(self): ... return self.a + self.b ... def __repr__(self): ... return f"{self.__class__.__name__}(a={self.a}, b={self.b})" >>> >>> assert isinstance(A, type) == isinstance(partialclass(A), type) == True >>> >>> assert str(signature(A)) == '(a=0, b=1)' >>> >>> a = A() >>> assert a.mysum() == 1 >>> assert str(a) == 'A(a=0, b=1)' >>> >>> assert A(a=10).mysum() == 11 >>> assert str(A()) == 'A(a=0, b=1)' >>> >>> >>> AA = partialclass(A, b=2) >>> assert str(signature(AA)) == '(a=0, *, b=2)' >>> aa = AA() >>> assert aa.mysum() == 2 >>> assert str(aa) == 'A(a=0, b=2)' >>> assert AA(a=1, b=3).mysum() == 4 >>> assert str(AA(3)) == 'A(a=3, b=2)' >>> >>> AA = partialclass(A, a=7) >>> assert str(signature(AA)) == '(*, a=7, b=1)' >>> assert AA().mysum() == 8 >>> assert str(AA(a=3)) == 'A(a=3, b=1)' ``` Note in the last partial that since `a` was fixed, you need to specify the keyword `AA(a=3)`. `AA(3)` won’t work: ```pycon >>> AA(3) Traceback (most recent call last): ... TypeError: __init__() got multiple values for argument 'a' ``` On the other hand, you can use `*args` to specify the fixtures: ```pycon >>> AA = partialclass(A, 22) >>> assert str(AA()) == 'A(a=22, b=1)' >>> assert str(signature(AA)) == '(b=1)' >>> assert str(AA(3)) == 'A(a=22, b=3)' ``` ### dol.util.read_from_bytes(file_reader, obj=None, \*, buffer_arg_position=0, buffer_arg_name=None, io_buffer_cls=, \*\*kwargs) Takes a file reading function that expects a file-like object, and returns a function that instead of reading from a file, reads from bytes. This is the read version of the `written_bytes` function of the same module. #### NOTE If obj is not given, read_from_bytes will return a “bytes reader” function that takes obj as the first argument, and uses the file_reader to read the bytes. * **Parameters:** * **file_reader** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – A function that reads from a file-like object. * **obj** ([`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes)) – The bytes to read. * **buffer_arg_position** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – The position of the file-like object in file_reader’s arguments. * **buffer_arg_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the file-like object argument in file_reader. * **Returns:** The result of reading from the bytes. Example usage: Using `json.load` to read a JSON object from bytes: ```pycon >>> import json >>> data = {'a': 1, 'b': 2} >>> json_bytes = json.dumps(data).encode('utf-8') >>> read_json_from_bytes = read_from_bytes(json.load) >>> data_loaded = read_json_from_bytes(json_bytes) >>> data_loaded == data True ``` Using `pickle.load` to read an object from bytes: ```pycon >>> import pickle >>> obj = {'x': [1, 2, 3], 'y': ('a', 'b')} >>> pickle_bytes = pickle.dumps(obj) >>> read_pickle_from_bytes = read_from_bytes(pickle.load) >>> obj_loaded = read_pickle_from_bytes(pickle_bytes) >>> obj_loaded == obj True ``` ### dol.util.regroupby(items, \*key_funcs, \*\*named_key_funcs) Recursive groupby. Applies the groupby function recursively, using a sequence of key functions. #### NOTE The named_key_funcs argument names don’t have any external effect. They just give a name to the key function, for code reading clarity purposes. #### SEE ALSO groupby, itertools.groupby, and dol.source.SequenceKvReader ```pycon >>> # group by how big the number is, then by it's mod 3 value >>> # note that named_key_funcs argument names doesn't have any external effect (but give a name to the function) >>> regroupby([1, 2, 3, 4, 5, 6, 7], lambda x: 'big' if x > 5 else 'small', mod3=lambda x: x % 3) {'small': {1: [1, 4], 2: [2, 5], 0: [3]}, 'big': {0: [6], 1: [7]}} >>> >>> tokens = ['the', 'fox', 'is', 'in', 'a', 'box'] >>> stopwords = {'the', 'in', 'a', 'on'} >>> word_category = lambda x: 'stopwords' if x in stopwords else 'words' >>> regroupby(tokens, word_category, len) {'stopwords': {3: ['the'], 2: ['in'], 1: ['a']}, 'words': {3: ['fox', 'box'], 2: ['is']}} >>> regroupby(tokens, len, word_category) {3: {'stopwords': ['the'], 'words': ['fox', 'box']}, 2: {'words': ['is'], 'stopwords': ['in']}, 1: {'stopwords': ['a']}} ``` ### dol.util.safe_compile(path, normalize_path=True) Compile a *literal file path* into a regex pattern that matches that path, normalizing separators and escaping regex-special characters on Windows. #### WARNING This is for **path templates only**, NOT for general regexes. It `re.escape`-s its argument on Windows, which turns any regex into a literal-string matcher there. To compile an actual regex, use `re.compile` (see `dol.trans.filter_regex`, fixed to do exactly that). Its output is intentionally platform-dependent (Windows paths get escaped), so callers must not rely on a specific `.pattern` across OSes. * **Parameters:** **path** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The file path to be compiled into a regex pattern. * **Returns:** A compiled regular expression object for the given path. * **Return type:** [*Pattern*](https://docs.python.org/3/library/re.html#re.Pattern) ### Examples ```pycon >>> import re >>> isinstance(safe_compile("/fun/paths/are/awesome"), re.Pattern) True >>> isinstance(safe_compile(r"C:\folder\file.txt"), re.Pattern) True ``` ### *class* dol.util.staticproperty(function) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A decorator for defining static properties in classes. ```pycon >>> class A: ... @staticproperty ... def foo(): ... return 2 >>> A.foo 2 >>> A().foo 2 ``` ### dol.util.str_to_var_str(s) Make a valid python variable string from the input string. Left untouched if already valid. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> str_to_var_str('this_is_a_valid_var_name') 'this_is_a_valid_var_name' >>> str_to_var_str('not valid #)*(&434') 'not_valid_______434' >>> str_to_var_str('99_ballons') '_99_ballons' ``` ### dol.util.truncate_string_with_marker(s, , left_limit=15, right_limit=15, middle_marker='...') Return a string with a limited length. If the string is longer than the sum of the left_limit and right_limit, the string is truncated and the middle_marker is inserted in the middle. If the string is shorter than the sum of the left_limit and right_limit, the string is returned as is. ```pycon >>> truncate_string_with_marker('1234567890') '1234567890' ``` But if the string is longer than the sum of the limits, it is truncated: ```pycon >>> truncate_string_with_marker('1234567890', left_limit=3, right_limit=3) '123...890' >>> truncate_string_with_marker('1234567890', left_limit=3, right_limit=0) '123...' >>> truncate_string_with_marker('1234567890', left_limit=0, right_limit=3) '...890' ``` If you’re using a specific parametrization of the function often, you can create a partial function with the desired parameters: ```pycon >>> from functools import partial >>> truncate_string = partial(truncate_string_with_marker, left_limit=2, right_limit=2, middle_marker='---') >>> truncate_string('1234567890') '12---90' >>> truncate_string('supercalifragilisticexpialidocious') 'su---us' ``` ### dol.util.written_bytes(file_writer, obj=None, \*, obj_arg_position_in_writer=0, io_buffer_cls=) Takes a file writing function that expects an object and a file-like object, and returns a function that instead of writing to a file, returns the bytes that would have been written. This is the write version of the `read_from_bytes` function of the same module. #### NOTE If obj is not given, `write_bytes` will return a “bytes writer” function that takes obj as the first argument, and uses the file_writer to write the bytes. * **Parameters:** * **file_writer** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`), `Union`[[`BytesIO`](https://docs.python.org/3/library/io.html#io.BytesIO), [`StringIO`](https://docs.python.org/3/library/io.html#io.StringIO)]], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – A function that writes an object to a file-like object. * **obj** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)) – The object to write. * **Returns:** The bytes that would have been written to a file. Use case: When you have a function that writes to files, and you want to get an equivalent function but that gives you what bytes or string WOULD have been written to a file, so you can better reuse (to write elsewhere, for example, or because you need to pipe those bytes to another function). Example usage: Yes, we have json.dumps to get the JSON string, but what if (like is often the case) you just have a function that writes to a file-like object, like the `json.dump(obj, fp)` function? You can use `written_bytes` to get a function that will act as `json.dumps` like so: ```pycon >>> import json >>> get_json_bytes = written_bytes(json.dump, io_buffer_cls=io.StringIO) >>> get_json_bytes({'a': 1, 'b': 2}) '{"a": 1, "b": 2}' ``` Here’s another example with pandas DataFrame.to_parquet: ```python import pandas as pd df = pd.DataFrame({'column1': [1, 2, 3], 'column2': ['A', 'B', 'C']}) # Get a function that converts DataFrame to Parquet bytes df_to_parquet_bytes = written_bytes(pd.DataFrame.to_parquet) # Get the bytes of the DataFrame in Parquet format parquet_bytes = df_to_parquet_bytes(df) all(pd.read_parquet(io.BytesIO(parquet_bytes)) == df) ``` ### dol.util.written_key(obj=None, writer=, \*, key=None, obj_arg_position_in_writer=0, encoder=) Writes an object to a key and returns the key. If key is not given, a temporary file is created and its path is returned. * **Parameters:** * **obj** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)) – The object to write. * **writer** (`Union`[[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – A function that writes an object to a file. * **key** (`Union`[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – The key (by default, filepath) to write to. If None, a temporary file is created. If a string starting with ‘\*’, the ‘\*’ is replaced with a unique temporary filename. If a string that has a ‘\*’ somewhere in the middle, what’s on the left of if is used as a directory and the ‘\*’ is replaced with a unique temporary filename. For example `'/tmp/*_file.ext'` would be replaced with `'/tmp/oiu8fj9873_file.ext'`. If a callable, it will be called with obj as input to get the key. One use case is to use a function that generates a key based on the object. * **obj_arg_position_in_writer** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Position of the object argument in writer function (0 or 1). * **encoder** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – A function that encodes the object before writing it. * **Returns:** The file path where the object was written. Example usage: Let’s make a store and a writer for that store. ```pycon >>> store = dict() >>> writer = writer=lambda obj, key: store.__setitem__(key, obj) ``` Note the order a writer expects is (obj, key), or we’d just be able to use `store.__setitem__` as our writer. If we specify a key, the object will be written to that key in the store and the key is output. ```pycon >>> written_key(42, writer=writer, key='my_key') 'my_key' >>> store {'my_key': 42} ``` Often, you’ll want to fix your writer (and possibly your key). You can do so with `functools.partial`, but for convenience, you can also just specify a writer, without an input object, and get a function that will write an object to a key. ```pycon >>> write_to_store = written_key(writer=writer, key='another_key') >>> write_to_store(99) 'another_key' >>> store {'my_key': 42, 'another_key': 99} ``` If you don’t specify a key, a temporary file is created and the key is output. ```pycon >>> write_to_store = written_key(writer=writer) >>> key = write_to_store(43) >>> key '/var/folders/mc/c070wfh51kxd9lft8dl74q1r0000gn/T/tmp8yaczd8b' >>> store[key] 43 ``` If the key you specify is a string with a ‘\*’, the ‘\*’ is replaced with a unique temporary filename, or the full path of the temporary file if the \* is at the start. ```pycon >>> write_to_store = written_key(writer=writer, key='*.ext') >>> key = write_to_store(44) >>> key '....ext' >>> store[key] 44 ``` One useful use case is when you want to pipe the output of one function into another function that expects a file path. What you need to do then is just pipe your written_key function into that function that expects to work with a file path, and it’ll be like piping the value of your input object into that function (just via a temp file). ```pycon >>> from dol.util import Pipe >>> store.clear() >>> key_func = lambda key: store.get(key) * 10 >>> pipe_obj_to_reader = Pipe(written_key(writer=writer), key_func) >>> pipe_obj_to_reader(45) 450 >>> store {...: 45} ``` The default writer is `write_to_file`, which can write bytes or strings to a file. If your object is not a bytes or string, you can specify an encoder to encode it before calling the writer. ```pycon >>> import json, pathlib >>> json_written_temp_filepath = written_key(key='*.json', encoder=json.dumps) >>> filepath = json_written_temp_filepath({'a': 1, 'b': 2}) >>> filepath '/var/folders/mc/c070wfh51kxd9lft8dl74q1r0000gn/T/tmp8yaczd8b.json' >>> json.loads(open(filepath).read()) {'a': 1, 'b': 2} ``` # _autosummary/dol.zipfiledol.html.md # dol.zipfiledol Data object layers and other utils to work with zip files. Main entry points: - `FilesOfZip`: read-only bytes of the files in a zip archive - `ZipReader`: same, but browsing folders as nested readers - `ZipFiles`: read-write-delete access to files in a zip archive - `FlatZipFilesReader`: the union of the contents of several zip files - `zip_compress`, `zip_decompress`: single-file zip bytes helpers ```pycon >>> from dol.zipfiledol import zip_compress, zip_decompress >>> zip_decompress(zip_compress(b'hello')) b'hello' ``` ### Functions | [`zip_compress`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.zip_compress)(b[, filename, compression, ...]) | Compress input bytes, returning the compressed bytes | |--------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| | [`zip_decompress`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.zip_decompress)(b, \*[, allowZip64, ...]) | Decompress input bytes of a single file zip, returning the uncompressed bytes | | [`to_zip_file`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.to_zip_file)(b, zip_filepath[, filename, ...]) | Zip input bytes and save to a single-file zip file. | | [`file_or_folder_to_zip_file`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.file_or_folder_to_zip_file)(src_path[, ...]) | Zip input bytes and save to a single-file zip file. | | [`if_i_zipped_stats`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.if_i_zipped_stats)(b) | Compress and decompress bytes with four different methods and return a dictionary of (size and time) stats. | | [`mk_flatzips_store`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.mk_flatzips_store)(dir_of_zips[, ...]) | A store so that you can work with a folder that has a bunch of zip files, as if they've all been extracted in the same folder. | | [`remove_some_entries_from_zip`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.remove_some_entries_from_zip)(zip_source, ...) | Removes specific keys from a zip file. | | [`remove_mac_junk_from_zip`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.remove_mac_junk_from_zip)(zip_source, \*[, ...]) | Removes mac junk keys from zip | ### Classes | `COMPRESSION`() | | |---------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------| | [`ZipReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipReader)(zip_file[, prefix, open_kws, ...]) | A KvReader to read the contents of a zip file. | | [`ZipInfoReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipInfoReader)(zip_file[, prefix, open_kws, ...]) | | | [`ZipFilesReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipFilesReader)(rootdir[, subpath, ...]) | A local file reader whose keys are the zip filepaths of the rootdir and values are corresponding ZipReaders. | | [`ZipFilesReaderAndBytesWriter`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipFilesReaderAndBytesWriter)(rootdir[, ...]) | Like ZipFilesReader, but the ability to write bytes (assumed to be valid bytes of the zip format) to a key | | [`FlatZipFilesReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.FlatZipFilesReader)(rootdir[, subpath, ...]) | Read the union of the contents of multiple zip files. | | [`FilesOfZip`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.FilesOfZip)(zip_file[, prefix, open_kws]) | | | [`FileStreamsOfZip`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.FileStreamsOfZip)(zip_file[, prefix, open_kws]) | Like FilesOfZip, but object returns are file streams instead. | | [`ZipFileStreamsReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipFileStreamsReader)(rootdir[, subpath, ...]) | Like ZipFilesReader, but objects returned are file streams instead. | | [`ZipStore`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipStore) | | | [`ZipFiles`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipFiles)(zip_filepath[, compression, ...]) | Zip read and writing. | ### Exceptions | [`OverwriteNotAllowed`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.OverwriteNotAllowed) | | |------------------------------------------------------------------------|----| | [`EmptyZipError`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.EmptyZipError) | | ### *exception* dol.zipfiledol.EmptyZipError Bases: [`KeyError`](https://docs.python.org/3/builtins/exceptions.html#KeyError), [`FileNotFoundError`](https://docs.python.org/3/builtins/exceptions.html#FileNotFoundError) ### *class* dol.zipfiledol.FileStreamsOfZip(zip_file, prefix='', open_kws=None) Bases: [`FilesOfZip`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.FilesOfZip) Like FilesOfZip, but object returns are file streams instead. So you use it like this: ```default z = FileStreamsOfZip(rootdir) with z[relpath] as fp: ... # do stuff with fp, like fp.readlines() or such... ``` ### *class* dol.zipfiledol.FilesOfZip(zip_file, prefix='', open_kws=None) Bases: [`ZipReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipReader) ### *class* dol.zipfiledol.FlatZipFilesReader(rootdir, subpath='.+\\\\.zip', pattern_for_field=None, max_levels=0, zip_reader=, \*\*zip_reader_kwargs) Bases: [`FlatReader`](_autosummary/dol.sources.html.md#dol.sources.FlatReader), [`ZipFilesReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipFilesReader) Read the union of the contents of multiple zip files. A local file reader whose keys are the zip filepaths of the rootdir and values are corresponding ZipReaders. Example use case: A remote data provider creates snapshots of whatever changed (modified files and new ones…) since the last snapshot, dumping snapshot zip files in a specic accessible location. You make `remote` and `local` stores and can update your local. Then you can perform syncing actions such as: ```python missing_keys = remote.keys() - local.keys() local.update({k: remote[k] for k in missing_keys}) # downloads missing snapshots ``` The data will look something like this: ```python dump_folder/ 2021_09_11.zip 2021_09_12.zip 2021_09_13.zip etc. ``` both on remote and local. What should then local do to use this data? Unzip and merge? Well, one solution, provided through FlatZipFilesReader, is to not unzip at all, but instead, give you a store that provides you a view “as if you unzipped and merged”. ### *exception* dol.zipfiledol.OverwriteNotAllowed Bases: [`FileExistsError`](https://docs.python.org/3/builtins/exceptions.html#FileExistsError), [`OverWritesNotAllowedError`](_autosummary/dol.errors.html.md#dol.errors.OverWritesNotAllowedError) ### *class* dol.zipfiledol.ZipFileStreamsReader(rootdir, subpath='.+\\\\.zip', pattern_for_field=None, max_levels=0, \*, zip_reader=, \*\*zip_reader_kwargs) Bases: [`PrefixRelativizationMixin`](_autosummary/dol.paths.html.md#dol.paths.PrefixRelativizationMixin), [`Store`](_autosummary/dol.base.html.md#dol.base.Store) Like ZipFilesReader, but objects returned are file streams instead. #### is_valid_key(k, \*args, \_\_name='is_valid_key', \*\*kwargs) `is_valid_key` on the inner key – see `mk_relative_path_store`. #### validate_key(k, \*args, \_\_name='validate_key', \*\*kwargs) `validate_key` on the inner key – see `mk_relative_path_store`. ### *class* dol.zipfiledol.ZipFiles(zip_filepath, compression=8, allow_overwrites=True, pwd=None) Bases: [`KvPersister`](_autosummary/dol.base.html.md#dol.base.KvPersister) Zip read and writing. When you want to read zips, there’s the `FilesOfZip`, `ZipReader`, or `ZipFilesReader` we know and love. Sometimes though, you want to write to zips too. For this, we have `ZipFiles`. Since ZipFiles can write to a zip, it’s read functionality is not going to assume static data, and cache things, as your favorite zip readers did. This, and the acrobatics need to disguise the weird zipfile into something more… key-value natural, makes for a not so efficient store, out of the box. I advise using one of the zip readers if all you need to do is read, or subclassing or wrapping ZipFiles with caching layers if it is appropriate to you. Let’s verify that a ZipFiles can indeed write data. First, we’ll set things up! ```pycon >>> from tempfile import gettempdir >>> import os >>> >>> rootdir = gettempdir() >>> >>> # preparation >>> test_zipfile = os.path.join(rootdir, 'zipstore_test_file.zip') >>> if os.path.isfile(test_zipfile): ... os.remove(test_zipfile) >>> assert not os.path.isfile(test_zipfile) ``` Okay, test_zipfile doesn’t exist (but will soon…) ```pycon >>> z = ZipFiles(test_zipfile) ``` See that the file still doesn’t exist (it will only be created when we start writing) ```pycon >>> assert not os.path.isfile(test_zipfile) >>> list(z) # z "is" empty (which makes sense?) [] ``` Now let’s write something interesting (notice, it has to be in bytes): ```pycon >>> z['foo'] = b'bar' >>> list(z) # now we have something in z ['foo'] >>> z['foo'] # and that thing is what we put there b'bar' ``` And indeed we have a zip file now: ```pycon >>> assert os.path.isfile(test_zipfile) ``` ### *class* dol.zipfiledol.ZipFilesReader(rootdir, subpath='.+\\\\.zip', pattern_for_field=None, max_levels=0, zip_reader=, \*\*zip_reader_kwargs) Bases: [`FileCollection`](_autosummary/dol.filesys.html.md#dol.filesys.FileCollection), [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) A local file reader whose keys are the zip filepaths of the rootdir and values are corresponding ZipReaders. ### *class* dol.zipfiledol.ZipFilesReaderAndBytesWriter(rootdir, subpath='.+\\\\.zip', pattern_for_field=None, max_levels=0, zip_reader=, \*\*zip_reader_kwargs) Bases: [`ZipFilesReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipFilesReader) Like ZipFilesReader, but the ability to write bytes (assumed to be valid bytes of the zip format) to a key ### *class* dol.zipfiledol.ZipInfoReader(zip_file, prefix='', , open_kws=None, file_info_filt=None) Bases: [`ZipReader`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipReader) ### *class* dol.zipfiledol.ZipReader(zip_file, prefix='', , open_kws=None, file_info_filt=None) Bases: [`KvReader`](_autosummary/dol.base.html.md#dol.base.KvReader) A KvReader to read the contents of a zip file. Provides a KV perspective of [https://docs.python.org/3/library/zipfile.html](https://docs.python.org/3/library/zipfile.html) `ZipReader` has two value categories: Directories and Files. Both categories are distinguishable by the keys, through the “ends with slash” convention. When a file, the value return is bytes, as usual. When a directory, the value returned is a `ZipReader` itself, with all params the same, except for the `prefix`, which serves to specify the subfolder (that is, `prefix` acts as a filter). #### NOTE If you get data zipped by a mac, you might get some junk along with it. Namely `__MACOSX` folders `.DS_Store` files. I won’t rant about it, since others have. But you might find it useful to remove them from view. One choice is to use `dol.trans.filt_iter` to get a filtered view of the zips contents. In most cases, this should do the job: ```default # applied to store instance or class: store = filt_iter(filt=lambda x: not x.startswith('__MACOSX') and '.DS_Store' not in x)(store) ``` Another option is just to remove these from the zip file once and for all. In unix-like systems: ```default zip -d filename.zip __MACOSX/\* zip -d filename.zip \*/.DS_Store ``` ### Examples ```default # >>> s = ZipReader('/path/to/some_zip_file.zip') # >>> len(s) # 53432 # >>> list(s)[:3] # the first 3 elements (well... their keys) # ['odir/', 'odir/app/', 'odir/app/data/'] # >>> list(s)[-3:] # the last 3 elements (well... their keys) # ['odir/app/data/audio/d/1574287049078391/m/Ctor.json', # 'odir/app/data/audio/d/1574287049078391/m/intensity.json', # 'odir/app/data/run/status.json'] # >>> # getting a file (note that by default, you get bytes, so need to decode) # >>> s['odir/app/data/run/status.json'].decode() # b'{"test_phase_number": 9, "test_phase": "TestActions.IGNORE_TEST", "session_id": 0}' # >>> # when you ask for the contents for a key that's a directory, # >>> # you get a ZipReader filtered for that prefix: # >>> s['odir/app/data/audio/'] # ZipReader('/path/to/some_zip_file.zip', 'odir/app/data/audio/', {}, ) # >>> # Often, you only want files (not directories) # >>> # You can filter directories out using the file_info_filt argument # >>> s = ZipReader('/path/to/some_zip_file.zip', file_info_filt=ZipReader.FILES_ONLY) # >>> len(s) # compare to the 53432 above, that contained dirs too # 53280 # >>> list(s)[:3] # first 3 keys are all files now # ['odir/app/data/plc/d/1574304926795633/d/1574305026895702', # 'odir/app/data/plc/d/1574304926795633/d/1574305276853053', # 'odir/app/data/plc/d/1574304926795633/d/1574305159343326'] # >>> # >>> # ZipReader.FILES_ONLY and ZipReader.DIRS_ONLY are just convenience filt functions # >>> # Really, you can provide any custom one yourself. # >>> # This filter function should take a ZipInfo object, and return True or False. # >>> # (https://docs.python.org/3/library/zipfile.html#zipfile.ZipInfo) # >>> # >>> import re # >>> p = re.compile('audio.*\.json$') # >>> my_filt_func = lambda fileinfo: bool(p.search(fileinfo.filename)) # >>> s = ZipReader('/Users/twhalen/Downloads/2019_11_21.zip', file_info_filt=my_filt_func) # >>> len(s) # 48 # >>> list(s)[:3] # ['odir/app/data/audio/d/1574333557263758/m/Ctor.json', # 'odir/app/data/audio/d/1574333557263758/m/intensity.json', # 'odir/app/data/audio/d/1574288084739961/m/Ctor.json'] ``` ### dol.zipfiledol.ZipStore alias of [`ZipFiles`](_autosummary/dol.zipfiledol.html.md#dol.zipfiledol.ZipFiles) ### dol.zipfiledol.file_or_folder_to_zip_file(src_path, zip_filepath=None, filename=None, , compression=8, allow_overwrites=True, pwd=None) Zip input bytes and save to a single-file zip file. ### dol.zipfiledol.if_i_zipped_stats(b) Compress and decompress bytes with four different methods and return a dictionary of (size and time) stats. ```pycon >>> b = b'x' * 1000 + b'y' * 1000 # 2000 (quite compressible) bytes >>> if_i_zipped_stats(b) {'uncompressed': {'bytes': 2000, 'comp_time': 0, 'uncomp_time': 0}, 'deflated': {'bytes': 137, 'comp_time': 0.00015592575073242188, 'uncomp_time': 0.00012612342834472656}, 'bzip2': {'bytes': 221, 'comp_time': 0.0013129711151123047, 'uncomp_time': 0.0011119842529296875}, 'lzma': {'bytes': 206, 'comp_time': 0.0058901309967041016, 'uncomp_time': 0.0005228519439697266}} ``` ### dol.zipfiledol.mk_flatzips_store(dir_of_zips, zip_pair_path_preproc=, mk_store=, \*\*extra_mk_store_kwargs) A store so that you can work with a folder that has a bunch of zip files, as if they’ve all been extracted in the same folder. Note that `zip_pair_path_preproc` can be used to control how to resolve key conflicts (i.e. when you get two different zip files that have a same path in their contents). The last path encountered by `zip_pair_path_preproc(zip_path_pairs)` is the one that will be used, so one should make `zip_pair_path_preproc` act accordingly. ### dol.zipfiledol.remove_mac_junk_from_zip(zip_source, \*, keys_to_be_removed=, ask_before_before_deleting=False, remove_action='filter') Removes mac junk keys from zip ### dol.zipfiledol.remove_some_entries_from_zip(zip_source, keys_to_be_removed, ask_before_before_deleting=True, , remove_action='filter') Removes specific keys from a zip file. * **Parameters:** * **zip_source** – zip filepath, bytes, or whatever a `ZipFiles` can take * **keys_to_be_removed** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)] | [`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – An iterable of keys or a boolean filter function * **ask_before_before_deleting** – True (default) if the user should be presented with the keys first, and asked permission to delete. * **Returns:** The ZipFiles (in case you want to do further work with it) #### TIP If you want to delete with no questions asked, use currying: ```pycon >>> from functools import partial >>> rm_keys_without_asking = partial( ... remove_some_entries_from_zip, ... ask_before_before_deleting=False ... ) ``` ### dol.zipfiledol.to_zip_file(b, zip_filepath, filename=None, , compression=8, allow_overwrites=True, pwd=None, encoding='utf-8') Zip input bytes and save to a single-file zip file. * **Parameters:** * **b** ([`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes) | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Input bytes or string * **zip_filepath** – zip filepath to save the zipped input to * **filename** – The name/path of the zip entry we want to save to * **encoding** – In case the input is str, the encoding to use to convert to bytes ### dol.zipfiledol.zip_compress(b, filename='some_bytes', , compression=8, allowZip64=True, compresslevel=None, strict_timestamps=True, encoding='utf-8') Compress input bytes, returning the compressed bytes * **Return type:** [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes) ```pycon >>> b = b'x' * 1000 + b'y' * 1000 # 2000 (quite compressible) bytes >>> len(b) 2000 >>> >>> zipped_bytes = zip_compress(b) >>> # Note: Compression details will be system dependent >>> len(zipped_bytes) 137 >>> unzipped_bytes = zip_decompress(zipped_bytes) >>> unzipped_bytes == b # verify that unzipped bytes are the same as the original True >>> >>> from dol.zipfiledol import compression_methods >>> >>> zipped_bytes = zip_compress(b, compression=compression_methods['bzip2']) >>> # Note: Compression details will be system dependent >>> len(zipped_bytes) 221 >>> unzipped_bytes = zip_decompress(zipped_bytes) >>> unzipped_bytes == b # verify that unzipped bytes are the same as the original True ``` ### dol.zipfiledol.zip_decompress(b, , allowZip64=True, compresslevel=None, strict_timestamps=True) Decompress input bytes of a single file zip, returning the uncompressed bytes See `zip_compress` for usage examples. * **Return type:** [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes) # about-this-build.html.md # About this build This documentation was built on **2026-09-22 13:56 UTC** from commit 66e3056 on branch master, for **dol 0.3.70** (from pyproject.toml). #### NOTE Nothing suggests a mismatch: the tree was clean at the commit above, and the documented version is the one on PyPI. ## Source | | | |---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | Commit | 66e30561fbea869c2e55fe5ab258f665b286ac71 | | Branch | master | | Tags at this commit | 0.3.70 | | Working tree | clean | | Remote | https://github.com/i2mint/dol | ## Continuous integration | | | |--------------|--------------------------------------------------------------------------------------------| | Repository | i2mint/dol | | Run | 35736360865 | | Ref | refs/heads/master | | Event commit | 6e71c71a053dce9d34325ad7cade5f0509a7cee3 (in the history of the built commit) | ## Tools | | | |----------|---------| | epythet | 0.2.12 | | Sphinx | 9.1.0 | | docutils | 0.22.4 | | Python | 3.12.14 | ## Configuration as resolved | | | |---------------|------------------------------------------------------------------| | theme | auto (Sphinx theme furo) | | accent | #694191 | | api_generator | autosummary | | ignore | tests/, scrap/, examples/ | | agent_outputs | true | | aggregates | md | | ai_artifacts | true | ## Package on PyPI Latest release: 0.3.70, the same as the documented version. ## Reproduce ```bash git clone https://github.com/i2mint/dol && cd dol git checkout 66e30561fbea869c2e55fe5ab258f665b286ac71 pip install "epythet==0.2.12" epythet quickstart . --ignore tests/ scrap/ examples/ ``` The same data, for machines: build_info.json (schema version 1). # ai-agents.html.md # For AI agents `dol` ships artifacts for coding agents alongside its code. This page lists them, says where each lives in the repository, and points at the machine-readable copies of this documentation. ## Skills Skills are folders holding a `SKILL.md` (the [Agent Skills](https://agentskills.io) format): a description that tells an agent when to use it and a body with the procedure. Install one into your agent with `gh skill` (any host: `--agent claude-code`, `copilot`, `cursor`, `codex`, `gemini`), or use the copy bundled in the wheel. ### `dol-dev-portability` Keep dol working on Windows as well as Linux/macOS. Use when touching dol’s path/key machinery (filesys.py, naming.py, paths.py, util.py), compiling regexes from templates/paths, mapping keys<->filesystem paths, or when a dol test/doctest passes on Linux/macOS but fails on the Windows CI job. Covers dol’s specific landmines: safe_compile is path-only (never compile a regex with it), escape template LITERALS not the whole pattern, os.sep consistency in prefix/affix codecs, empty-prefix handling, POSIX-only os calls, and dol’s native-separator key convention. For general cross-platform Python principles, see the global `cross-platform-python` skill. Source: [`.claude/skills/dol-dev-portability`](https://github.com/i2mint/dol/tree/HEAD/.claude/skills/dol-dev-portability). ### `dol-dev-wrap-kvs` Understand and safely modify dol’s core wrapping machinery — wrap_kvs, store_decorator, Store.wrap, and how transforms are applied. Use when touching dol/trans.py or dol/base.py; when changing how key/value transforms (key_of_id, id_of_key, obj_of_data, data_of_obj, postget, preset) are called; when a transform passed to wrap_kvs behaves unexpectedly (called with/without the store as first arg); when working Issues #9/#12/#18/#6/#5; or when a wrap_kvs change could ripple through the ~32 ecosystem packages that use it. Covers the signature-conditioning rule (name AND arity), the FirstArgIsMapping marker, the delegation (has-a) architecture and its ‘self is unwrapped’ trap, the subclass-signature-freeze trap, and the mandatory dependents test-gate. For end-user store-building, see the consumer skills; for Windows path issues, see dol-dev-portability. Source: [`.claude/skills/dol-dev-wrap-kvs`](https://github.com/i2mint/dol/tree/HEAD/.claude/skills/dol-dev-wrap-kvs). ### `dol-store-building` Build a dol store: wrap any storage backend (files, S3, DB, dict, an API) behind a uniform dict-like (MutableMapping) interface, adding key and value transforms/serialization. Use when a user wants to give a backend a dict interface, add JSON/pickle/gzip (or custom) serialization to a store, transform or filter keys, compose codecs, cache a slow store, or asks ‘how do I use dol to …’. Covers wrap_kvs (the core), the ValueCodecs/KeyCodecs namespaces, Pipe composition, the ready-made file stores (Files/TextFiles/JsonFiles/PickleFiles), filt_iter, the test-with-dict-then-swap-backend workflow, and self-aware transforms via FirstArgIsMapping. For authoring interactive scaffolds see the /new-store, /add-codec, /explain-store commands; for modifying dol’s internals see dol-dev-wrap-kvs. Source: [`.claude/skills/dol-store-building`](https://github.com/i2mint/dol/tree/HEAD/.claude/skills/dol-store-building). ## Instruction files Files agents read before working in this repository. - [`CLAUDE.md`](https://github.com/i2mint/dol/tree/HEAD/CLAUDE.md): read by Claude Code ## Machine-readable documentation This site publishes the same documentation in forms that fit an agent’s context window: - [`llms.txt`](https://i2mint.github.io/dol/llms.txt): an index of every page with a one-line description ([llms.txt](https://llmstxt.org) format) - [`dol.md`](https://i2mint.github.io/dol/dol.md): the whole documentation as one Markdown file - `.html.md`: a rendered Markdown twin of every page, advertised from each page’s `` with `` - [`objects.inv`](https://i2mint.github.io/dol/objects.inv): the Sphinx inventory: a symbol-to-URL index (`sphobjinv convert plain objects.inv -`) # api.html.md # API reference | [`dol`](_autosummary/dol.html.md#module-dol) | Core tools to build simple interfaces to complex data sources and bend the interface to your will (and need). | |-------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|