of a zip. We'll give these to FilesOfZip to be able to read them
First 4 file names in the zip: ['AK.TXT', 'AL.TXT', 'AR.TXT', 'AZ.TXT']
```
|
state |
gender |
year |
name |
number |
| 0 |
AK |
F |
1910 |
Mary |
14 |
| 1 |
AK |
F |
1910 |
Annie |
12 |
| 2 |
AK |
F |
1910 |
Anna |
10 |
| 3 |
AK |
F |
1910 |
Margaret |
8 |
| 4 |
AK |
F |
1910 |
Helen |
7 |
| ... |
... |
... |
... |
... |
... |
| 28957 |
AK |
M |
2019 |
Patrick |
5 |
| 28958 |
AK |
M |
2019 |
Ronin |
5 |
| 28959 |
AK |
M |
2019 |
Sterling |
5 |
| 28960 |
AK |
M |
2019 |
Titus |
5 |
| 28961 |
AK |
M |
2019 |
Tucker |
5 |
28962 rows × 5 columns
### Example using emoji image urls data
```python
url = 'https://raw.githubusercontent.com/thorwhalen/my_sources/master/github_emojis.json'
if url in g: # if we've cached this already
del g[url] # remove it from cache
assert url not in g
```
```python
import json
d = json.loads(g[url].decode())
len(d)
```
```none
1510
```
```python
list(d)[330:340]
```
```none
['couple_with_heart_woman_man',
'couple_with_heart_woman_woman',
'couplekiss_man_man',
'couplekiss_man_woman',
'couplekiss_woman_woman',
'cow',
'cow2',
'cowboy_hat_face',
'crab',
'crayon']
```
```python
d['cow']
```
```none
'https://github.githubassets.com/images/icons/emoji/unicode/1f42e.png?v8'
```
### A little py2store exercise: A store to get image objects of emojis
As a demo of py2store, let’s make a store that allows you to get (displayable) image objects of emojis, taking care of downloading and caching
the name:url information for you.
```python
from functools import cached_property
import json
from py2store import KvReader
from graze import graze
class EmojiUrls(KvReader):
"""A store of emoji urls. Will automatically download and cache emoji (name, url) map to a local file when first used."""
data_source_url = 'https://raw.githubusercontent.com/thorwhalen/my_sources/master/github_emojis.json'
@cached_property
def data(self):
b = graze(self.data_source_url) # does the same thing as Graze()[url]
return json.loads(b.decode())
def __iter__(self):
yield from self.data
def __getitem__(self, k):
return self.data[k]
# note, normally you would define an explicit __len__ and __contains__ to make these more efficient
emojis = EmojiUrls()
len(emojis), emojis['cow']
```
```none
(1510,
'https://github.githubassets.com/images/icons/emoji/unicode/1f42e.png?v8')
```
```python
from IPython.display import Image
import requests
from py2store import wrap_kvs, add_ipython_key_completions
@add_ipython_key_completions # this enables tab-completion of keys in jupyter notebooks
@wrap_kvs(obj_of_data=lambda url: Image(requests.get(url).content))
class EmojiImages(EmojiUrls):
"""An emoji reader returning Image objects (displayable in jupyter notebooks)"""
emojis = EmojiImages()
len(emojis)
```
```none
1510
```
```python
emojis['cow']
```
## Grub
Quick and easy search engine of anything (that can be expressed as a key-value store of text).
### search your code
```python
# Make a store to search in (only requirements is that it provide text values)
import os
import py2store
rootdir = os.path.dirname(py2store.__file__)
store_to_search = LocalBinaryStore(os.path.join(rootdir) + '{}.py') # The '{}.py' is a short-hand of LocalBinaryStore to filter for .py files only
# make a search object for that store
from grub import SearchStore
search = SearchStore(store_to_search)
```
```python
search('cache key-value pairs')
```
```none
array(['py2store/caching.py', 'py2store/utils/cumul_aggreg_write.py',
'py2store/trans.py', 'py2store/examples/write_caches.py',
'py2store/utils/cache_descriptors.py',
'py2store/utils/explicit.py',
'py2store/persisters/arangodb_w_pyarango.py',
'py2store/persisters/dynamodb_w_boto3.py',
'py2store/stores/delegation_stores.py', 'py2store/util.py'],
dtype=object)
```
### search jokes (and download them automatically
Some code that acquires and locally caches a joke data, makes a mapping view of it (here just a `dict` in memory), and builds a search engine to find jokes. All that, in a few lines.
```python
import json
from graze.base import graze
from grub import SearchStore
# reddit jokes (194553 at the time of writing this)
jokes_url = 'https://raw.githubusercontent.com/taivop/joke-dataset/master/reddit_jokes.json'
raw_data = json.loads(graze(jokes_url).decode())
joke_store = {x['id']: f"{x['title']}\n--> {x['body']}\n(score: {x['score']})" for x in raw_data}
search_joke = SearchStore(joke_store)
```
```python
results_idx = search_joke('searching for something funny')
print(joke_store[results_idx[0]]) # top joke (not by score, but by relevance to search terms)
```
```none
want to hear me say something funny?
--> well alright then...."something funny" there
(score: 0)
```
# More examples
## Looks like a dict
Below, we make a default store and demo a few basic operations on it.
The default store uses a dict as it’s backend persister.
A dict is neither really a backend, nor a persister. But it helps to try things out with no
footprint.
```python
from py2store.base import Store
s = Store()
assert list(s) == []
s['foo'] = 'bar' # put 'bar' in 'foo'
assert 'foo' in s # check that 'foo' is in (i.e. a key of) s
assert s['foo'] == 'bar' # see that the value that 'foo' contains is 'bar'
assert list(s) == ['foo'] # list all the keys (there's only one)
assert list(s.items()) == [('foo', 'bar')] # list all the (key, value) pairs
assert list(s.values()) == ['bar'] # list all the values
assert len(s) == 1 # Number of items in my store
s['another'] = 'item' # store another item
assert len(s) == 2 # Now I have two!
assert list(s) == ['foo', 'another'] # here they are
```
There’s nothing fantastic in the above code.
I’ve just demoed some operations on a dict.
But it’s exactly this simplicity that py2store aims for.
You can now replace the `s = Store()` with `s = AnotherStore(...)` where `AnotherStore`
now uses some other backend that could be remote or local, could be a database, or any
system that can store `something` (the value) `somewhere` (the key).
You can choose from an existing store (e.g. local files, for AWS S3, for MongoDB) or
quite easily make your own (more on that later).
And yet, it will still look like you’re talking to a dict. This not only means that you can
talk to various storage systems without having to actually learn how to, but also means
that the same business logic code you’ve written can be reused with no modification.
But py2store offers more than just a simple consistent facade to **where** you store things,
but also provides means to define **how** you do it.
In the case of key-value storage, the “how” is defined on the basis of the keys (how you reference)
the objects you’re storing and the values (how you serialize and deserialize those objects).
## Converting keys: Relative paths and absolute paths
Take a look at the following example, that adds a layer of key conversion to a store.
```python
# defining the store
from py2store.base import Store
class PrefixedKeyStore(Store):
prefix = ''
def _id_of_key(self, key):
return self.prefix + key # prepend prefix before passing on to store
def _key_of_id(self, _id):
if not _id.startswith(self.prefix):
raise ValueError(f"_id {_id} wasn't prefixed with {self.prefix}")
else:
return _id[len(self.prefix):] # don't show the user the prefix
# trying the store out
s = PrefixedKeyStore()
s.prefix = '/ROOT/'
assert list(s) == []
s['foo'] = 'bar' # put 'bar' in 'foo'
assert 'foo' in s # check that 'foo' is in (i.e. a key of) s
assert s['foo'] == 'bar' # see that the value that 'foo' contains is 'bar'
assert list(s) == ['foo'] # list all the keys (there's only one)
assert list(s.items()) == [('foo', 'bar')] # list all the (key, value) pairs
assert list(s.values()) == ['bar'] # list all the values
assert len(s) == 1 # Number of items in my store
s['another'] = 'item' # store another item
assert len(s) == 2 # Now I have two!
assert list(s) == ['foo', 'another'] # here they are
```
Q: That wasn’t impressive! It’s just the same as the first Store. What’s this prefix all about?
A: The prefix thing is hidden, and that’s the point. You want to talk the “relative” (i.e “prefix-free”)
language, but may have the need for this prefix to be prepended to the key before persisting the data
and that prefix to be removed before being displayed to the user.
Think of working with files. Do you want to have to specify the root folder every time you store something
or retrieve something?
Q: Prove it!
A: Okay, let’s look under the hood at what the underlying store (a dict) is dealing with:
```python
assert list(s.store.items()) == [('/ROOT/foo', 'bar'), ('/ROOT/another', 'item')]
```
You see? The keys that the “backend” is using are actually prefixed with `"/ROOT/"`
## Serialization/Deserialization
Let’s now demo serialization and deserialization.
Say we want to deserialize any text we stored by appending `"hello "` to everything stored.
```python
# defining the store
from py2store.base import Store
class MyFunnyStore(Store):
def _obj_of_data(self, data):
return f'hello {data}'
# trying the store out
s = MyFunnyStore()
assert list(s) == []
s['foo'] = 'bar' # put 'bar' in 'foo'
assert 'foo' in s # check that 'foo' is in (i.e. a key of) s
assert s['foo'] == 'hello bar' # the value that 'foo' contains SEEMS to be 'hello bar'
assert list(s) == ['foo'] # list all the keys (there's only one)
assert list(s.items()) == [('foo', 'hello bar')] # list all the (key, value) pairs
assert list(s.values()) == ['hello bar'] # list all the values
```
Note: This is an easy example to demo on-load transformation of data (i.e. deserialization),
but wouldn’t be considered “deserialization” by all.
See the [Should storage transform the data?]() discussion below.
In the following, we want to serialize our text by upper-casing it (and see it as such)
when we retrieve the text.
```python
# defining the store
from py2store.base import Store
class MyOtherFunnyStore(Store):
def _data_of_obj(self, obj):
return obj.upper()
# trying the store out
s = MyOtherFunnyStore()
assert list(s) == []
s['foo'] = 'bar' # put 'bar' in 'foo'
assert 'foo' in s # check that 'foo' is in (i.e. a key of) s
assert s['foo'] == 'BAR' # see that the value that 'foo' contains is 'bar'
assert list(s) == ['foo'] # list all the keys (there's only one)
assert list(s.items()) == [('foo', 'BAR')] # list all the (key, value) pairs
assert list(s.values()) == ['BAR'] # list all the values
```
In the last to serialization examples, we only implemented one way transformations.
That’s all fine if you just want to have a writer (so only need a serializer) or a reader (so only
need a deserializer).
In most cases though, you will need two way transformations, specifying how the object
should be serialized to be stored, and how it should be deserialized to get your object back.
## A pickle store
Say you wanted the store to pickle as your serializer. Here’s how this could look like.
```python
# defining the store
import pickle
from py2store.base import Store
class PickleStore(Store):
protocol = None
fix_imports = True
encoding = 'ASCII'
def _data_of_obj(self, obj): # serializer
return pickle.dumps(obj, protocol=self.protocol, fix_imports=self.fix_imports)
def _obj_of_data(self, data): # deserializer
return pickle.loads(data, fix_imports=self.fix_imports, encoding=self.encoding)
# trying the store out
s = PickleStore()
assert list(s) == []
s['foo'] = 'bar' # put 'bar' in 'foo'
assert s['foo'] == 'bar' # I can get 'bar' back
# behind the scenes though, it's really a pickle that is stored:
assert s.store['foo'] == b'\x80\x03X\x03\x00\x00\x00barq\x00.'
```
Again, it doesn’t seem that impressive that you can get back a string that you stored in a dict.
For two reasons: (1) you don’t really need to serialize strings to store them and (2) you don’t need to serialize python
objects to store them in a dict.
But if you (1) were trying to store more complex types and (2) were actually persisting them in a file system or database,
then you’ll need to serialize.
The point here is that the serialization and persisting concerns are separated from the storage and retrieval concern.
The code still looks like you’re working with a dict.
## But how do you change the persister?
By using a persister that persists where you want.
You can also write your own. All a persister needs to work with py2store is that it follows the interface
python’s `collections.MutableMapping` (or a subset thereof). More on how to make your own persister later
You just need to follow the collections.MutableMapping interface.
Below a simple example of how to persist in files under a given folder.
(Warning: If you want a local file store, don’t use this, but one of the easier to use, robust and safe stores in the
stores folder!)
```python
import os
from collections.abc import MutableMapping
class SimpleFilePersister(MutableMapping):
"""Read/write (text or binary) data to files under a given rootdir.
Keys must be absolute file paths.
Paths that don't start with rootdir will be raise a KeyValidationError
"""
def __init__(self, rootdir, mode='t'):
if not rootdir.endswith(os.path.sep):
rootdir = rootdir + os.path.sep
self.rootdir = rootdir
assert mode in {'t', 'b', ''}, f"mode ({mode}) not valid: Must be 't' or 'b'"
self.mode = mode
def __getitem__(self, k):
with open(k, 'r' + self.mode) as fp:
data = fp.read()
return data
def __setitem__(self, k, v):
with open(k, 'w' + self.mode) as fp:
fp.write(v)
def __delitem__(self, k):
os.remove(k)
def __contains__(self, k):
""" Implementation of "k in self" check.
Note: MutableMapping gives you this for free, using a try/except on __getitem__,
but the following uses faster os functionality."""
return os.path.isfile(k)
def __iter__(self):
yield from filter(os.path.isfile,
map(lambda x: os.path.join(self.rootdir, x),
os.listdir(self.rootdir)))
def __len__(self):
"""Note: There's system-specific faster ways to do this."""
count = 0
for _ in self.__iter__():
count += 1
return count
def clear(self):
"""MutableMapping creates a 'delete all' functionality by default. Better disable it!"""
raise NotImplementedError("If you really want to do that, loop on all keys and remove them one by one.")
```
Now try this out:
```python
import os
# What folder you want to use. Defaulting to the home folder. You can choose another place, but make sure
rootdir = os.path.expanduser('~/') # Defaulting to the home folder. You can choose another place
persister = SimpleFilePersister(rootdir)
foo_fullpath = os.path.join(rootdir, 'foo')
persister[foo_fullpath] = 'bar' # write 'bar' to a file named foo_fullpath
assert persister[foo_fullpath] == 'bar' # see that you can read the contents of that file to get your 'bar' back
assert foo_fullpath in persister # the full filepath indeed exists in (i.e. "is a key of") the persister
assert foo_fullpath in list(persister) # you can list all the contents of the rootdir and file foo_fullpath in it
```
## Talk your own CRUD dialect
Don’t like this dict-like interface? Want to talk **your own** CRUD words?
We got you covered! Just subclass `SimpleFilePersister` and make the changes you want to make:
```python
class MySimpleFilePersister(SimpleFilePersister):
# If it's just renaming, it's easy
read = SimpleFilePersister.__getitem__
exists = SimpleFilePersister.__contains__
n_files = SimpleFilePersister.__len__
# here we want a new method that gives us an actual list of the filepaths in the rootdir
list_files = lambda self: list(self.__iter__())
# And for write we want val and key to be swapped in our interface,
def write(self, val, key): # note that we wanted val to come first here (as with json.dump and pickle.dump interface)
return self.__setitem__(key, val)
my_persister = MySimpleFilePersister(rootdir)
foo_fullpath = os.path.join(rootdir, 'foo1')
my_persister.write('bar1', foo_fullpath) # write 'bar1' to a file named foo_fullpath
assert my_persister.read(foo_fullpath) == 'bar1' # see that you can read the contents of that file to get your 'bar1' back
assert my_persister.exists(foo_fullpath) # the full filepath indeed exists in (i.e. "is a key of") the persister
assert foo_fullpath in my_persister.list_files() # you can list all the contents of the rootdir and file foo_fullpath in it
```
## Transforming keys
But dealing with full paths can be annoying, and might couple code too tightly with a particular local system.
We’d like to use relative paths instead.
Easy: Wrap the persister in the `PrefixedKeyStore` defined earlier.
```python
s = PrefixedKeyStore(store=persister) # wrap your persister with the PrefixedKeyStore defined earlier
if not rootdir.endswith(os.path.sep):
rootdir = rootdir + os.path.sep # make sure the rootdir ends with slash
s.prefix = rootdir # use rootdir as prefix in keys
s['foo2'] = 'bar2' # write 'bar2' to a file
assert s['foo2'] == 'bar2' # see that you can read the contents of that file to get your 'bar2' back
assert 'foo2' in s
assert 'foo2' in list(s)
```
# How it works
py2store offers three aspects that you can define or modify to store things where you like and how you like it:
* **Persistence**: Where things are actually stored (memory, files, DBs, etc.)
* **Serialization**: Value transformaton.
How python objects should be transformed before it is persisted,
and how persisted data should be transformed into python objects.
* **Indexing**: Key transformation. How you name/id/index your data.
Full or relative paths. Unique combination of parameters (e.g. (country, city)). Etc.
All of this allows you to do operations such as “store this (value) in there (persitence) as that (key)”,
moving the tedious particularities of the “in there” as well how the “this” and “that” are transformed to fit
in there, all out of the way of the business logic code. The way it should be.
Note: Where data is actually persisted just depends on what the base CRUD methods
(`__getitem__`, `__setitem__`, `__delitem__`, `__iter__`, etc.) define them to be.
# A few persisters you can use
We’ll go through a few basic persisters that are ready to use.
There are more in each category, and we’ll be adding new categories, but
this should get you started.
Here is a useful function to perform a basic test on a store, given a key and value.
It doesn’t test all store method (see test modules for that), but demos
the basic functionality that pretty much every store should be able to do.
```python
def basic_test(store, k='foo', v='bar'):
""" This test performs
Warning: Don't use on a key k that you don't want to loose!"""
if k in store: # deleting all docs in tmp
del store[k]
assert (k in store) == False # see that key is not in store (and testing __contains__)
orig_length = len(store) # the length of the store before insertion
store[k] = v # write v to k (testing __setitem__)
assert store[k] == v # see that the value can be retrieved (testing __getitem__, and that __setitem__ worked)
assert len(store) == orig_length + 1 # see that the number of items in the store increased by 1
assert (k in store) == True # see that key is in store now (and testing __contains__ again)
assert k in list(store) # testing listing the (key) contents of a store (and seeing if )
assert store.get(k) == v # the get method
_ = next(iter(store.keys())) # get the first key (test keys method)
_ = next(iter(store.__iter__())) # get the first key (through __iter__)
k in store.keys() # test that the __contains__ of store.keys() works
try:
_ = next(iter(store.values())) # get the first value (test values method)
_ = next(iter(store.items())) # get the first (key, val) pair (test items method)
except Exception:
print("values() (therefore items()) didn't work: Probably testing a persister that had other data in it that your persister doesn't like")
assert (k in store) == True # testing __contains__ again
del store[k] # clean up (and test delete)
```
## Local Files
There are many choices of local file stores according to what you’re trying to do.
One general (but not too general) purpose local file store is
‘py2store.stores.local_store.RelativePathFormatStoreEnforcingFormat’.
It can do a lot for you, like add a prefix to your keys (so you can talk in relative instead of absolute paths),
lists all files in subdirectories as well recursively,
only show you files that have a given pattern when you list them,
and not allow you to write to a key that doesn’t fit the pattern.
Further, it also has what it takes to create parametrized paths or parse out the parameters of a path.
```python
from py2store.stores.local_store import RelativePathFormatStoreEnforcingFormat as LocalFileStore
import os
rootdir = os.path.expanduser('~/pystore_tests/') # or replace by the folder you want to use
os.makedirs(rootdir, exist_ok=True) # this will make all directories that don't exist. Don't use if you don't want that.
store = LocalFileStore(path_format=rootdir)
basic_test(store, k='foo', v='bar')
```
The signature of LocalFileStore is:
```python
LocalFileStore(path_format, mode='',
buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)
```
Often path_format is just used to specify the rootdir, as above.
But you can specify the desired format further.
For example, the following will only yield .wav files,
and only allow you to write to keys that end with .wav:
```python
store = LocalFileStore(path_format='/THE/ROOT/DIR/{}.wav')
```
The following will additional add the restriction that those .wav files have the format ‘SOMESTRING_’
followed by digits:
```python
store = LocalFileStore(path_format='/THE/ROOT/DIR/{:s}_{:d}.wav')
```
You get the point…
The other arguments of LocalFileStore or more or less those of python’s `open` function.
The slight difference is that here the `mode` argument applies both to read and write.
If `mode='b'` for example, the file will be opened with `mode='rb'` when opened to read and
with `mode='wb'` when opened to write. For assymetrical read/write modes, the
user can specify a `read_mode` and `write_mode` (in this case the `mode` argument is ignored).
## MongoDB
A MongoDB collection is not as naturally a key-value storage as a file system is.
MongoDB stores “documents”, which are JSONs of data, having many (possibly nested) fields that are not
by default enforced by a schema. So in order to talk to mongo as a key-value store, we need to
specify what fields should be considered as keys, and what fields should be considered as data.
By default, the `_id` field (the only field ensured by default to contain unique values) is the single key field, and
all other fields are considered to be data fields.
Note: py2store mongo tools have now been moved to the mongodol project. Import from there.
Requires `pymongo`.
```python
from mongodol.stores import MongoStore # Note: project moved to mongodol now
# The following makes a default MongoStore, the default pymongo.MongoClient settings,
# and db_name='py2store', collection_name='test', key_fields=('_id',)
store = MongoStore()
basic_test(store, k={'_id': 'foo'}, v={'val': 'bar', 'other_val': 3})
```
But it can get annoying to specify the key as a dict every time.
The key schema is fixed, so you should be able to just specify the tuple of values making the keys.
And you can, with MongoTupleKeyStore
```python
from mongodol.stores import MongoTupleKeyStore # Note: project moved to mongodol now
store = MongoTupleKeyStore(key_fields=('_id', 'name'))
basic_test(store, k=(1234, 'bob'), v={'age': 42, 'gender': 'unspecified'})
```
## S3, SQL, Zips, Dropbox
S3 persister/stores work pretty much like LocalStores, but stores in S3. You’ll need to have an account with
AWS to use this. Find S3 stores in py2store.stores.s3_stores.
SQL give you read and write access to SQL DBs and tables.
ZipReader (and other related stores) talks to one or several files, giving you the ability to operate as if the zips were uncompressed.
Dropbox will give you access to dropbox files through the same dict-like interface.
# Miscellenous
## Caching
There’s some basic caching capabilities in py2store.
Basic, but covers a lot of use cases.
But if you want to bring your own caching tools, you might be able to use them here too.
For example, the very popular `cachetools` uses a `dict` as it’s default cache store, but you can
specify any mutable mapping (that takes tuples as keys!).
Say you want to use local files as your cache. Try something like this:
```python
from cachetools import cached # there's also LRUCache, TTLCache...
from py2store import QuickPickleStore, wrap_kvs
def tuple_to_str(k: tuple, sep: str=os.path.sep) -> str:
return sep.join(k)
if isinstance(k, tuple):
return os.path.sep.join(k)
else:
return k
def str_to_tuple(k: str, sep: str=os.path.sep) -> tuple:
return k.split(sep)
@wrap_kvs(id_of_key=tuple_to_str, key_of_id=str_to_tuple)
class TupledQuickPickleStore(QuickPickleStore):
"""A local pickle store with tuple keys (to work well with cachetools)"""
local_files_cache = TupledQuickPickleStore() # no rootdir? Fine, will choose a local file
@cached(cache=local_files_cache)
def hello(x='world'):
return f"hello {x}!"
```
```pydocstring
>>> hello('QT')
>>> import pickle
>>> # Let's now verify that we actually have a file with such content
>>> with open(os.path.join(local_files_cache._prefix, 'QT'), 'rb') as fp:
... file_contents = pickle.load(fp)
>>> assert file_contents == 'hello QT!'
```
# Philosophical FAQs
## Is a store an ORM? A DAO?
Call it what you want, really.
It would be tempting to coin py2store as ya(p)orm (yet another (python) object-relational mapping),
but that would be misleading. The intent of py2store is not to map objects to db entries,
but rather to offer a consistent interface for basic storage operations.
In that sense, py2store is more akin to an implementation of the data access object (DAO) pattern.
Of course, the difference between ORM and DAO can be blurry, so all this should be taken with a grain of salt.
Advantages and disadvantages such abstractions are easy to search and find, but in most cases the
pros probably outweigh the cons.
Most data interaction mechanisms can be satisfied by a subset of the collections.abc interfaces.
For example, one can use python’s collections.Mapping interface for any key-value storage, making the data access
object have the look and feel of a dict, instead of using other popular method name choices such for
such as read/write, load/dump, etc.
One of the dangers there is that, since the DAO looks and acts like a dict (but is not) a user might underestimate
the running-costs of some operations.
## Should storage transform the data?
When does “storing data” **not** transform data? The answer is that storage almost always transforms data in some way.
But some of these transformations are taken for granted, because there’s so often “attached”
(i.e. “co-occur”) with the raw process of storing. In py2store, the data transformation is attached to (but not entangled with) the store object.
This means you have a specific place where you can check or change that aspect of storage.
Having a consistent and simple interface to storage is useful. Being able to attach key and value
transformations to this interface is also very useful. But though you get a lot for cheap, it’s
not free: Mapping the many (storage systems operations) to the one (consistent interface) means
that, through habit, you might project some misaligned expectations.
This is one of the known disadvantages of Data Access Objects (DAOs))
Have a look at this surreal behavior:
```python
# defining the store
from py2store.base import Store
class MyFunnyStore(Store):
def _obj_of_data(self, data):
return f'hello {data}'
# trying the store out
s = MyFunnyStore()
s['foo'] = 'bar' # put 'bar' in 'foo'
assert s['foo'] == 'hello bar' # the value that 'foo' contains SEEMS to be 'hello bar'
# so look how surreal that can be:
s['foo'] = s['foo'] # retrieve what's under 'foo' and store it back into 'foo'
assert s['foo'] == 'hello hello bar' # what the...
s['foo'] = s['foo'] # retrieve what's under 'foo' and store it back into 'foo'
assert s['foo'] == 'hello hello hello bar' # No no no! I do not like green eggs and ham!
```
This happens, because though you’ve said `s['foo'] = 'bar'`, the value returned by `s['foo']` is
actually `'hello bar'`. Why? Because though you’ve stored `'bar'`, you’re transforming the data when you
read it (that’s what `_obj_of_data` does).
Is that a desirable behavior? Transforming the stored data before handing it to the user?
Well, this is such a common pattern that it has it’s own acronym and tools named after the acronym: ETL.
Extract, Transform, Load.
What is happening here is that we composed extraction and transformation. Is that acceptable?
Say I have a big store of tagged audio files of various formats but only want to work with
files containing the ‘gunshot’ tag and lasting no more than 10s, and further get the data as a
waveform (a sequence of samples).
You’d probably find this acceptable:
```python
audio_file_type=type_of(file)
with open(file, 'wb') as fp:
file_bytes = fp.read()
wf = convert_to_waveform(file_bytes)
```
Or this:
```python
filt = mk_file_filter(tag='gunshot', max_size_s=10)
for file in filter(filt, audio_source):
with open(file, 'wb') as fp:
file_bytes = fp.read()
wf = convert_to_waveform(file_bytes, audio_file_type=type_of(file))
send_wf_for_analysis(wf)
```
You might even find it acceptable to put such code in a functions called `get_waveform_from_file`,
or `generator_of_waveforms_of_filtered_files`.
So why is it harder to accept something where you make a store that encompasses your needs.
You do `s = WfStore(audio_source, filt)` and then
```python
wf = s[some_file] # get a waveform
```
or
```python
for wf in s.values(): # iterate over all waveforms
send_wf_for_analysis(wf)
```
It’s harder to accept precisely because of the simplicity and consistency (with dict operations).
We’re used to `s[some_file]` meaning “give me THE value stored in s, in the ‘some_file’ slot”.
We’re not used to `s[some_file]` meaning
“go get the data stored in `some_file` and give it to me in a format more convenient for my use”.
Stores allow you to compose extraction and transformation, or transformation and loading,
and further specifying filter, caching, indexing, and many other aspects related to storage.
Those, py2store helps you create the perspective you want, or need.
That said, one needs to be careful that the simplicity thus created doesn’t induce misuse.
For example, in the `MyFunnyStore` example above, we may want to use a different store to persist
and to read, and perhaps reflect their function in their names. For example:
```python
# defining the store
from py2store.base import Store
class ExtractAndTransform(Store):
def _obj_of_data(self, data):
return f'hello {data}'
store = Store()
extract_and_transform = ExtractAndTransform(store)
store['foo'] = 'bar' # put 'bar' in 'foo'
assert store['foo'] == 'bar' # the value that store contains for 'foo' is 'bar'
assert extract_and_transform['foo'] == 'hello bar' # the value that extract_and_transform gives you is 'bar'
```
# Some links
Presentation at PyBay 2019: https://www.youtube.com/watch?v=6lx0A6oVM5E
ETL: Extract, Transform, Load: https://en.wikipedia.org/wiki/Extract,_transform,_load
ORM: Object-relational mapping: https://en.wikipedia.org/wiki/Object-relational_mapping
DAO: Data access object: https://en.wikipedia.org/wiki/Data_access_object
DRY: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself
SOC: Separation Of Concerns: https://en.wikipedia.org/wiki/Separation_of_concerns
COC: Convention Over Configuration: https://en.wikipedia.org/wiki/Convention_over_configuration
This documentation as a single file: py2store.md (Markdown, for agents).
# _autosummary/py2store.access.html.md
# py2store.access
Utils to load stores from store specifications.
Includes the logic to allow configurations (and defaults) to be parametrized by external environmental
variables and files.
Every data-sourced problem has it’s problem-relevant stores. Once you get your stores right, along with the
right access credentials, indexing, serialization, caching, filtering etc. you’d like to be able to name, save
and/or share this specification, and easily get access to it later on.
Here are tools to help you out.
There are two main key-value stores: One for configurations the user wants to reuse, and the other for the user’s
desired defaults. Both have the same structure:
> * first level key: Name of the resource (should be a valid python variable name)
> * The reminder is more or less free form (until the day we lay out some schemas for this)
The system will look for the specification of user_configs and user_defaults in a json file.
The filepath to this json file can specified in environment variables
> PY2STORE_CONFIGS_JSON_FILEPATH and PY2STORE_DEFAULTS_JSON_FILEPATH
respectively.
By default, they are:
```default
~/.py2store_configs.json and ~/.py2store_defaults.json
```
respectively.
### Functions
| [`add_json_ext`](_autosummary/py2store.access.html.md#py2store.access.add_json_ext)(k) | Append `.json` to `k`. |
|--------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`assert_callable`](_autosummary/py2store.access.html.md#py2store.access.assert_callable)(f) | Return `f` unchanged if it is callable, else raise `AssertionError`. |
| [`compose`](_autosummary/py2store.access.html.md#py2store.access.compose)(\*functions) | Make a function that is the composition of the input functions |
| [`dflt_func_loader`](_autosummary/py2store.access.html.md#py2store.access.dflt_func_loader)(f) | Loads and returns the function referenced by f, which could be a callable or a DOTPATH_TO_MODULE.FUNC_NAME dotpath string to one, or a pipeline of these |
| [`directory_json_items`](_autosummary/py2store.access.html.md#py2store.access.directory_json_items)() | Yield `(name, contents)` for every `.json` file in the user configs directory, warning about files that fail to decode. |
| [`dotpath_to_func`](_autosummary/py2store.access.html.md#py2store.access.dotpath_to_func)(f) | Loads and returns the function referenced by f, which could be a callable or a DOTPATH_TO_MODULE.FUNC_NAME dotpath string to one. |
| [`dotpath_to_obj`](_autosummary/py2store.access.html.md#py2store.access.dotpath_to_obj)(dotpath) | Loads and returns the object referenced by the string DOTPATH_TO_MODULE.OBJ_NAME |
| [`fakit`](_autosummary/py2store.access.html.md#py2store.access.fakit)(fak[, func_loader]) | Execute a fak with given f, a, k and function loader. |
| [`fakit_from_dict`](_autosummary/py2store.access.html.md#py2store.access.fakit_from_dict)(d[, func_loader]) | Call the function in `d['f']` (through `func_loader`) with the args `d['a']` and kwargs `d['k']`, both optional. |
| [`fakit_from_tuple`](_autosummary/py2store.access.html.md#py2store.access.fakit_from_tuple)(t[, func_loader]) | Call the function in `t[0]` (through `func_loader`) with the args and kwargs in the rest of `t`. |
| [`getenv`](_autosummary/py2store.access.html.md#py2store.access.getenv)(name[, default]) | Like os.getenv, but removes a suffix r character if present (problem with some env var systems) |
| [`mkdir_if_needed`](_autosummary/py2store.access.html.md#py2store.access.mkdir_if_needed)(dirpath[, name, verbose]) | Create `dirpath` if it does not exist, printing a note that calls it `name` (`verbose` is accepted but not used). |
| [`without_json_ext`](_autosummary/py2store.access.html.md#py2store.access.without_json_ext)(_id) | Strip the `.json` extension from `_id`. |
### Classes
| [`MyConfigs`](_autosummary/py2store.access.html.md#py2store.access.MyConfigs)([max_levels]) | |
|----------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| [`MyStores`](_autosummary/py2store.access.html.md#py2store.access.MyStores)([store]) | Store specifications (json files under the user configs directory) that instantiate the store on read. |
### *class* py2store.access.MyConfigs(max_levels=None)
Bases: `OverWritesNotAllowedMixin`, [`MyConfigs`](_autosummary/py2store.access.html.md#py2store.access.MyConfigs)
### *class* py2store.access.MyStores(store=)
Bases: `Store`
Store specifications (json files under the user configs directory) that instantiate the store on read.
A specification is a dict with a `'$fak'` entry holding an `(f, a, k)` specification, run through `fakit`.
#### *property* configs
The underlying store of raw specifications.
#### *static* func_loader()
Loads and returns the function referenced by f,
which could be a callable or a DOTPATH_TO_MODULE.FUNC_NAME dotpath string to one, or a pipeline of these
* **Return type:**
`callable`
### py2store.access.add_json_ext(k)
Append `.json` to `k`.
### py2store.access.assert_callable(f)
Return `f` unchanged if it is callable, else raise `AssertionError`.
* **Return type:**
`callable`
### py2store.access.compose(\*functions)
Make a function that is the composition of the input functions
### py2store.access.dflt_func_loader(f)
Loads and returns the function referenced by f,
which could be a callable or a DOTPATH_TO_MODULE.FUNC_NAME dotpath string to one, or a pipeline of these
* **Return type:**
`callable`
### py2store.access.directory_json_items()
Yield `(name, contents)` for every `.json` file in the user configs directory, warning about files that fail to decode.
### py2store.access.dotpath_to_func(f)
Loads and returns the function referenced by f,
which could be a callable or a DOTPATH_TO_MODULE.FUNC_NAME dotpath string to one.
* **Return type:**
`callable`
### py2store.access.dotpath_to_obj(dotpath)
Loads and returns the object referenced by the string DOTPATH_TO_MODULE.OBJ_NAME
### py2store.access.fakit(fak, func_loader=)
Execute a fak with given f, a, k and function loader.
Essentially returns `func_loader(f)(*a, **k)`
* **Parameters:**
* **fak** – A (f, a, k) specification. Could be a tuple or a dict (with ‘f’, ‘a’, ‘k’ keys). All but f are optional.
* **func_loader** – A function returning a function. This is where you specify any validation of func specification f,
and/or how to get a callable from it.
* **Returns:**
A python object.
### py2store.access.fakit_from_dict(d, func_loader=)
Call the function in `d['f']` (through `func_loader`) with the args `d['a']` and kwargs `d['k']`, both optional.
### py2store.access.fakit_from_tuple(t, func_loader=)
Call the function in `t[0]` (through `func_loader`) with the args and kwargs in the rest of `t`.
`t` has 1 to 3 elements: `(f,)`, `(f, args)`, `(f, kwargs)` or `(f, args, kwargs)`,
where `args` is a tuple or list and `kwargs` a dict.
```pycon
>>> fakit_from_tuple((len, ['abc']))
3
>>> fakit_from_tuple(('builtins.len', ['ab']))
2
>>> fakit_from_tuple((dict, (), {'x': 1}))
{'x': 1}
```
### py2store.access.getenv(name, default=None)
Like os.getenv, but removes a suffix r character if present (problem with some env var systems)
### py2store.access.mkdir_if_needed(dirpath, name=None, verbose=True)
Create `dirpath` if it does not exist, printing a note that calls it `name` (`verbose` is accepted but not used).
### py2store.access.without_json_ext(\_id)
Strip the `.json` extension from `_id`.
# _autosummary/py2store.appendable.html.md
# py2store.appendable
### py2store.appendable(store_cls=None, , item2kv, return_keys=False, \_\_module_\_=None, \_\_name_\_=None, \_\_qualname_\_=None, \_\_doc_\_=None, \_\_annotations_\_=None, \_\_defaults_\_=None, \_\_kwdefaults_\_=None)
Makes a new class with append (and consequential extend) methods
* **Parameters:**
* **store_cls** – The store class to subclass
* **item2kv** – The function that produces a (key, val) pair from an item
* **new_store_name** – The name to give the new class (default will be ‘Appendable’ + store_cls._\_name_\_)
* **Returns:**
append, and extend.
* **Return type:**
A subclass of store_cls with two additional methods
```pycon
>>> item_to_kv = lambda item: (item['L'], item) # use value of 'L' as the key, and value is the item itself
>>> MyStore = appendable(dict, item2kv=item_to_kv)
>>> s = MyStore(); s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items())
[('let', {'L': 'let', 'I': 'it', 'G': 'go'})]
```
Use mk_item2kv.from_item_to_key_params_and_val with tuple key params
```pycon
>>> item_to_kv = appendable.mk_item2kv_for.item_to_key_params_and_val(lambda x: ((x['L'], x['I']), x['G']), '{}/{}')
>>> MyStore = appendable(item2kv=item_to_kv)(dict) # showing the append(...)(store) form
>>> s = MyStore(); s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items())
[('let/it', 'go')]
```
Use mk_item2kv.from_item_to_key_params_and_val with dict key params
```pycon
>>> item_to_kv = appendable.mk_item2kv_for.item_to_key_params_and_val(
... lambda x: ({'L': x['L'], 'G': x['G']}, x['I']), '{G}_{L}')
>>> @appendable(item2kv=item_to_kv) # showing the @ form
... class MyStore(dict):
... pass
>>> s = MyStore(); s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items())
[('go_let', 'it')]
```
Use mk_item2kv.fields to get a tuple key from item fields,
defining the sub-dict of the remaining fields to be the value.
Also showing here how you can decorate the instance itself.
```pycon
>>> item_to_kv = appendable.mk_item2kv_for.fields(['G', 'L'], key_as_tuple=True)
>>> d = {}
>>> s = appendable(d, item2kv=item_to_kv)
>>> s.append({'L': 'let', 'I': 'it', 'G': 'go'}); list(s.items())
[(('go', 'let'), {'I': 'it'})]
```
You can make the “append” and “extend” methods to return the new generated keys by
using the “return_keys” flag.
```pycon
>>> d = {}
>>> s = appendable(d, item2kv=item_to_kv, return_keys=True)
>>> s.append({'L': 'let', 'I': 'it', 'G': 'go'})
('go', 'let')
```
# _autosummary/py2store.base.html.md
# py2store.base
Forwards to dol.base:
Base classes for making stores.
In the language of the collections.abc module, a store is a MutableMapping that is configured to work with a specific
representation of keys, serialization of objects (python values), and persistence of the serialized data.
That is, stores offer the same interface as a dict, but where the actual implementation of writes, reads, and listing
are configurable.
Consider the following example. You’re store is meant to store waveforms as wav files on a remote server.
Say waveforms are represented in python as a tuple (wf, sr), where wf is a list of numbers and sr is the sample
rate, an int). The \_\_setitem_\_ method will specify how to store bytes on a remote server, but you’ll need to specify
how to SERIALIZE (wf, sr) to the bytes that constitute that wav file: \_data_of_obj specifies that.
You might also want to read those wav files back into a python (wf, sr) tuple. The \_\_getitem_\_ method will get
you those bytes from the server, but the store will need to know how to DESERIALIZE those bytes back into a python
object: \_obj_of_data specifies that
Further, say you’re storing these .wav files in /some/folder/on/the/server/, but you don’t want the store to use
these as the keys. For one, it’s annoying to type and harder to read. But more importantly, it’s an irrelevant
implementation detail that shouldn’t be exposed. THe \_id_of_key and \_key_of_id pair are what allow you to
add this key interface layer.
These key converters object serialization methods default to the identity (i.e. they return the input as is).
This means that you don’t have to implement these as all, and can choose to implement these concerns within
the storage methods themselves.
# _autosummary/py2store.caching.html.md
# py2store.caching
Forwards to dol.caching:
Tools to add caching layers to stores.
# _autosummary/py2store.dig.html.md
# py2store.dig
Forwards to dol.dig:
Layers introspection
# _autosummary/py2store.errors.html.md
# py2store.errors
Forwards to dol.errors:
Error objects and utils
# _autosummary/py2store.ext.dataframes.html.md
# py2store.ext.dataframes
Data as `pandas.DataFrame` from various sources
# _autosummary/py2store.ext.docx.html.md
# py2store.ext.docx
Simple access to docx (Word Doc) elements.
# _autosummary/py2store.ext.github.html.md
# py2store.ext.github
a data object layer for github
# _autosummary/py2store.ext.gitlab.html.md
# py2store.ext.gitlab
Stores to talk to gitlab, using requests.
For example:
```default
ogl = GitLabAccessor(base_url="http://...", project_name=None)
print(ogl.get_project_names()) # prints all project names
ogl.set_project("PROJECT_NAME") # sets the project to "PROJECT_NAME"
print(
ogl.get_branch_names()
) # gets the branch names of current project (as set previously)
print(
ogl.get_branch("master")
) # gets a json of information about the master branch of current project.
```
# _autosummary/py2store.ext.hdf.html.md
# py2store.ext.hdf
a data object layer for HDF files
# _autosummary/py2store.ext.html.md
# py2store.ext
py2store Extensions, Add-ons, etc.
We kept py2store purely dependency-less, using only built-ins for everything but storage system connectors.
That said, in order to provide the user with more power, and show him/her how py2store tools can be used to build
powerful data accessors, we provide specialized modules that do require more than builtins. These dependencies are
not listed in the setup.py module, but we wrap their imports with informative ImportError handlers.
### Modules
| [`dataframes`](_autosummary/py2store.ext.dataframes.html.md#module-py2store.ext.dataframes) | Data as `pandas.DataFrame` from various sources |
|----------------------------------------------------------------------------------------------|---------------------------------------------------|
| [`docx`](_autosummary/py2store.ext.docx.html.md#module-py2store.ext.docx) | Simple access to docx (Word Doc) elements. |
| [`github`](_autosummary/py2store.ext.github.html.md#module-py2store.ext.github) | a data object layer for github |
| [`gitlab`](_autosummary/py2store.ext.gitlab.html.md#module-py2store.ext.gitlab) | Stores to talk to gitlab, using requests. |
| [`hdf`](_autosummary/py2store.ext.hdf.html.md#module-py2store.ext.hdf) | a data object layer for HDF files |
| [`matlab`](_autosummary/py2store.ext.matlab.html.md#module-py2store.ext.matlab) | a data object layer for matlab |
| [`wordnet`](_autosummary/py2store.ext.wordnet.html.md#module-py2store.ext.wordnet) | This moved to lexis project |
# _autosummary/py2store.ext.matlab.html.md
# py2store.ext.matlab
a data object layer for matlab
# _autosummary/py2store.ext.wordnet.html.md
# py2store.ext.wordnet
This moved to lexis project
The py2store wrapper to nltk.corpus.wordnet. Your no fuss gateway to (English) words.
The easiest way to get nltk.corpus.wordnet is
```text
pip install nltk
```
in your terminal, and then in a python console:
```default
import nltk; nltk.download('wordnet')
```
If you don’t like that way, [see here](https://www.nltk.org/install.html) for other ways to get wordnet.
The central construct of this module is the Synset (a set of synonyms that share a common meaning).
To see a few things you can do with Synsets, naked, [see here](https://www.nltk.org/howto/wordnet.html).
Here we put a py2store wrapper around this stuff.
What is WordNet? [https://wordnet.princeton.edu/](https://wordnet.princeton.edu/)
# _autosummary/py2store.filesys.html.md
# py2store.filesys
Forwards to dol.filesys:
File system access
# _autosummary/py2store.html.md
# py2store
py2store: tools to create simple and consistent interfaces to complicated and varied data sources.
The core has moved to the `dol` package (Data Object Layer); py2store keeps the original
names, re-exports them, and keeps the local-file stores that still live here. A store is a
`MutableMapping` whose keys and values are transformed on the way in and out, so that files,
zip archives or databases are read and written like a `dict`.
Main entry points:
- `LocalTextStore`, `LocalBinaryStore`, `LocalPickleStore`, `LocalJsonStore`: the files under a root directory as a dict
- `QuickStore`: the pickle store with a temporary default root and directories created on write
- `wrap_kvs`, `filt_iter`, `cached_keys`: transform a store’s keys, values or iteration (from `dol.trans`)
- `kvhead`, `ihead`: peek at the first items of a store or an iterable
```pycon
>>> from py2store import kvhead
>>> kvhead({'a': 1, 'b': 2})
('a', 1)
```
### Functions
| [`ihead`](_autosummary/py2store.html.md#py2store.ihead)(store[, n]) | Get the first item of an iterable, or a list of the first `n` items. |
|----------------------------------------------------------------------|---------------------------------------------------------------------------------|
| [`kvhead`](_autosummary/py2store.html.md#py2store.kvhead)(store[, n]) | Get the first `(key, value)` item of a store, or a list of the first `n` items. |
### py2store.ihead(store, n=1)
Get the first item of an iterable, or a list of the first `n` items.
With `n=1` the item itself is returned (`None` if the iterable is empty); otherwise a
list of at most `n` items.
```pycon
>>> ihead(iter('abc'))
'a'
>>> ihead('abc', 2)
['a', 'b']
>>> ihead(iter('')) is None
True
```
### py2store.kvhead(store, n=1)
Get the first `(key, value)` item of a store, or a list of the first `n` items.
With `n=1` the item itself is returned (`None` if the store is empty); otherwise a
list of at most `n` items, in the store’s iteration order.
```pycon
>>> kvhead({'a': 1, 'b': 2})
('a', 1)
>>> kvhead({'a': 1, 'b': 2}, 5)
[('a', 1), ('b', 2)]
>>> kvhead({}) is None
True
```
### Modules
| [`access`](_autosummary/py2store.access.html.md#module-py2store.access) | Utils to load stores from store specifications. |
|----------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|
| [`appendable`](_autosummary/py2store.appendable.html.md#py2store.appendable)([store_cls, return_keys, ...]) | Makes a new class with append (and consequential extend) methods |
| [`base`](_autosummary/py2store.base.html.md#module-py2store.base) | Forwards to dol.base: |
| [`caching`](_autosummary/py2store.caching.html.md#module-py2store.caching) | Forwards to dol.caching: |
| [`dig`](_autosummary/py2store.dig.html.md#module-py2store.dig) | Forwards to dol.dig: |
| [`errors`](_autosummary/py2store.errors.html.md#module-py2store.errors) | Forwards to dol.errors: |
| [`ext`](_autosummary/py2store.ext.html.md#module-py2store.ext) | py2store Extensions, Add-ons, etc. |
| [`filesys`](_autosummary/py2store.filesys.html.md#module-py2store.filesys) | Forwards to dol.filesys: |
| [`key_mappers`](_autosummary/py2store.key_mappers.html.md#module-py2store.key_mappers) | key mapping |
| [`misc`](_autosummary/py2store.misc.html.md#module-py2store.misc) | Functions to read from and write to misc sources. |
| [`mixins`](_autosummary/py2store.mixins.html.md#module-py2store.mixins) | Forwards to dol.mixins: |
| [`my`](_autosummary/py2store.my.html.md#module-py2store.my) | functionalities meant to be configurable |
| [`naming`](_autosummary/py2store.naming.html.md#module-py2store.naming) | Forwards to dol.naming: |
| [`parse_format`](_autosummary/py2store.parse_format.html.md#module-py2store.parse_format) | Modified from [https://github.com/r1chardj0n3s/parse](https://github.com/r1chardj0n3s/parse) |
| [`paths`](_autosummary/py2store.paths.html.md#module-py2store.paths) | Forwards to dol.paths: |
| [`persisters`](_autosummary/py2store.persisters.html.md#module-py2store.persisters) | base persisters -- now all forwarding to separate libraries |
| [`serializers`](_autosummary/py2store.serializers.html.md#module-py2store.serializers) | a package of serializers |
| [`signatures`](_autosummary/py2store.signatures.html.md#module-py2store.signatures) | Forwards to dol.signatures: |
| [`slib`](_autosummary/py2store.slib.html.md#module-py2store.slib) | Data Object Layers for a few standard libs. |
| [`sources`](_autosummary/py2store.sources.html.md#module-py2store.sources) | Forwards to dol.sources: |
| [`stores`](_autosummary/py2store.stores.html.md#module-py2store.stores) | a package of various stores |
| [`test`](_autosummary/py2store.test.html.md#module-py2store.test) | test files |
| [`trans`](_autosummary/py2store.trans.html.md#module-py2store.trans) | Forwards to dol.trans: |
| [`util`](_autosummary/py2store.util.html.md#module-py2store.util) | Forwards to dol.util: |
| [`utils`](_autosummary/py2store.utils.html.md#module-py2store.utils) | general utils |
# _autosummary/py2store.key_mappers.html.md
# py2store.key_mappers
key mapping
### Modules
| [`naming`](_autosummary/py2store.key_mappers.naming.html.md#module-py2store.key_mappers.naming) | This module only forwards to py2store.naming, and is deprecated. |
|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|
| [`paths`](_autosummary/py2store.key_mappers.paths.html.md#module-py2store.key_mappers.paths) | Module that forwards to py2store.paths, kept for back-compatibility |
| [`str_utils`](_autosummary/py2store.key_mappers.str_utils.html.md#module-py2store.key_mappers.str_utils) | utils from strings |
| [`tuples`](_autosummary/py2store.key_mappers.tuples.html.md#module-py2store.key_mappers.tuples) | Tools to map tuple-structured keys. |
# _autosummary/py2store.key_mappers.naming.html.md
# py2store.key_mappers.naming
This module only forwards to py2store.naming, and is deprecated.
# _autosummary/py2store.key_mappers.paths.html.md
# py2store.key_mappers.paths
Module that forwards to py2store.paths, kept for back-compatibility
# _autosummary/py2store.key_mappers.str_utils.html.md
# py2store.key_mappers.str_utils
utils from strings
### Functions
| [`args_and_kwargs_indices`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.args_and_kwargs_indices)(format_string) | Get the sets of indices and names used in manual specification of format strings, or None, None if auto spec. |
|-----------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
| [`auto_field_format_str`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.auto_field_format_str)(format_str) | Get an auto field version of the format_str |
| [`compile_str_from_parsed`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.compile_str_from_parsed)(parsed) | The (quasi-)inverse of string.Formatter.parse. |
| `empty_arg_and_kwargs_for_format`(format_string) | |
| [`format_params_in_str_format`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.format_params_in_str_format)(format_string) | Get the "parameter" indices/names of the format_string |
| [`get_explicit_positions`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.get_explicit_positions)(parsed_str_format) | |
| [`is_automatic_format_params`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.is_automatic_format_params)(format_params) | Says if the format_params is from an automatic specification |
| [`is_automatic_format_string`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.is_automatic_format_string)(format_string) | Says if the format_string is uses automatic specification |
| [`is_hybrid_format_params`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.is_hybrid_format_params)(format_params) | Says if the format_params is from a hybrid of auto and manual. |
| [`is_hybrid_format_string`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.is_hybrid_format_string)(format_string) | Says if the format_params is from a hybrid of auto and manual. |
| [`is_manual_format_params`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.is_manual_format_params)(format_params) | Says if the format_params is from a manual specification |
| [`is_manual_format_string`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.is_manual_format_string)(format_string) | Says if the format_string uses a manual specification |
| [`manual_field_format_str`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.manual_field_format_str)(format_str) | Get an auto field version of the format_str |
| [`n_format_params_in_str_format`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.n_format_params_in_str_format)(format_string) | The number of parameters |
| [`name_fields_in_format_str`](_autosummary/py2store.key_mappers.str_utils.html.md#py2store.key_mappers.str_utils.name_fields_in_format_str)(format_str[, ...]) | Get a manual field version of the format_str |
| `parse_str_format`(str_format) | |
| `transform_format_str`(format_str, ...) | |
### py2store.key_mappers.str_utils.args_and_kwargs_indices(format_string)
Get the sets of indices and names used in manual specification of format strings, or None, None if auto spec.
* **Parameters:**
**format_string** – A format string (i.e. a string with {…} to mark parameter placement and formatting
* **Returns:**
None, None if format_string is an automatic specification
set_of_indices_used, set_of_fields_used if it is a manual specification
```pycon
>>> format_string = '{0} (no 1) {2}, {see} this, {0} is a duplicate (appeared before) and {name} is string-named'
>>> assert args_and_kwargs_indices(format_string) == ({0, 2}, {'name', 'see'})
>>> format_string = 'This is a format string with only automatic field specification: {}, {}, {} etc.'
>>> assert args_and_kwargs_indices(format_string) == (set(), set())
```
### py2store.key_mappers.str_utils.auto_field_format_str(format_str)
Get an auto field version of the format_str
* **Parameters:**
**format_str** – A format string
* **Returns:**
A transformed format_str that has no names {inside} {formatting} {braces}.
```pycon
>>> auto_field_format_str('R/{0}/{one}/{}/{two}/T')
'R/{}/{}/{}/{}/T'
```
### py2store.key_mappers.str_utils.compile_str_from_parsed(parsed)
The (quasi-)inverse of string.Formatter.parse.
* **Parameters:**
* **parsed** – iterator of (literal_text, field_name, format_spec, conversion) tuples,
* **string.Formatter.parse** (*as yield by*)
* **Returns:**
A format string that would produce such a parsed input.
```pycon
>>> s = "ROOT/{}/{0!r}/{1!i:format}/hello{:0.02f}TAIL"
>>> assert compile_str_from_parsed(string.Formatter().parse(s)) == s
>>>
>>> # Or, if you want to see more details...
>>> parsed = list(string.Formatter().parse(s))
>>> for p in parsed:
... print(p)
('ROOT/', '', '', None)
('/', '0', '', 'r')
('/', '1', 'format', 'i')
('/hello', '', '0.02f', None)
('TAIL', None, None, None)
>>> compile_str_from_parsed(parsed)
'ROOT/{}/{0!r}/{1!i:format}/hello{:0.02f}TAIL'
```
### py2store.key_mappers.str_utils.format_params_in_str_format(format_string)
Get the “parameter” indices/names of the format_string
* **Parameters:**
**format_string** – A format string (i.e. a string with {…} to mark parameter placement and formatting
* **Returns:**
A list of parameter indices used in the format string, in the order they appear, with repetition.
Parameter indices could be integers, strings, or None (to denote “automatic field numbering”.
```pycon
>>> format_string = '{0} (no 1) {2}, and {0} is a duplicate, {} is unnamed and {name} is string-named'
>>> format_params_in_str_format(format_string)
[0, 2, 0, None, 'name']
```
### py2store.key_mappers.str_utils.get_explicit_positions(parsed_str_format)
```pycon
>>> parsed = parse_str_format("all/{}/is/{2}/position/{except}{this}{0}")
>>> get_explicit_positions(parsed)
{0, 2}
```
### py2store.key_mappers.str_utils.is_automatic_format_params(format_params)
Says if the format_params is from an automatic specification
#### SEE ALSO
is_manual_format_params and is_hybrid_format_params
### py2store.key_mappers.str_utils.is_automatic_format_string(format_string)
Says if the format_string is uses automatic specification
#### SEE ALSO
is_manual_format_params
```pycon
>>> is_automatic_format_string('Manual: indices: {1} {2}, named: {named} {fields}')
False
>>> is_automatic_format_string('Auto: only un-indexed and un-named: {} {}...')
True
>>> is_automatic_format_string('Hybrid: at least a {}, and a {0} or a {name}')
False
>>> is_manual_format_string('No formatting is both manual and automatic formatting!')
True
```
### py2store.key_mappers.str_utils.is_hybrid_format_params(format_params)
Says if the format_params is from a hybrid of auto and manual.
#### NOTE
Hybrid specifications are considered non-valid and can’t be formatted with format_string.format(…).
Yet, it can be useful for flexibility of expression (but will need to be resolved to be used).
#### SEE ALSO
is_manual_format_params and is_automatic_format_params
### py2store.key_mappers.str_utils.is_hybrid_format_string(format_string)
Says if the format_params is from a hybrid of auto and manual.
#### NOTE
Hybrid specifications are considered non-valid and can’t be formatted with format_string.format(…).
Yet, it can be useful for flexibility of expression (but will need to be resolved to be used).
```pycon
>>> is_hybrid_format_string('Manual: indices: {1} {2}, named: {named} {fields}')
False
>>> is_hybrid_format_string('Auto: only un-indexed and un-named: {} {}...')
False
>>> is_hybrid_format_string('Hybrid: at least a {}, and a {0} or a {name}')
True
>>> is_manual_format_string('No formatting is both manual and automatic formatting (so hybrid is both)!')
True
```
### py2store.key_mappers.str_utils.is_manual_format_params(format_params)
Says if the format_params is from a manual specification
#### SEE ALSO
is_automatic_format_params
### py2store.key_mappers.str_utils.is_manual_format_string(format_string)
Says if the format_string uses a manual specification
#### SEE ALSO
is_automatic_format_string and
```pycon
>>> is_manual_format_string('Manual: indices: {1} {2}, named: {named} {fields}')
True
>>> is_manual_format_string('Auto: only un-indexed and un-named: {} {}...')
False
>>> is_manual_format_string('Hybrid: at least a {}, and a {0} or a {name}')
False
>>> is_manual_format_string('No formatting is both manual and automatic formatting!')
True
```
### py2store.key_mappers.str_utils.manual_field_format_str(format_str)
Get an auto field version of the format_str
* **Parameters:**
**format_str** – A format string
* **Returns:**
A transformed format_str that has no names {inside} {formatting} {braces}.
```pycon
>>> auto_field_format_str('R/{0}/{one}/{}/{two}/T')
'R/{}/{}/{}/{}/T'
```
### py2store.key_mappers.str_utils.n_format_params_in_str_format(format_string)
The number of parameters
### py2store.key_mappers.str_utils.name_fields_in_format_str(format_str, field_names=None)
Get a manual field version of the format_str
* **Parameters:**
* **format_str** – A format string
* **names** – An iterable that produces enough strings to fill all of format_str fields
* **Returns:**
A transformed format_str
```pycon
>>> name_fields_in_format_str('R/{0}/{one}/{}/{two}/T')
'R/{0}/{1}/{2}/{3}/T'
>>> # Note here that we use the field name to inject a field format as well
>>> name_fields_in_format_str('R/{foo}/{0}/{}/T', ['42', 'hi:03.0f', 'world'])
'R/{42}/{hi:03.0f}/{world}/T'
```
# _autosummary/py2store.key_mappers.tuples.html.md
# py2store.key_mappers.tuples
Tools to map tuple-structured keys.
That is, converting from any of the following kinds of keys:
> * tuples (or list-like)
> * dicts
> * formatted/templated strings
> * dsv (Delimiter-Separated Values)
### Functions
| `dict_of_str`(d, compiled_regex) | |
|------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|
| `dict_of_tuple`(d, fields) | |
| [`dsv_of_list`](_autosummary/py2store.key_mappers.tuples.html.md#py2store.key_mappers.tuples.dsv_of_list)(d[, sep]) | Converting a list of strings to a dsv (delimiter-separated values) string. |
| [`list_of_dsv`](_autosummary/py2store.key_mappers.tuples.html.md#py2store.key_mappers.tuples.list_of_dsv)(d[, sep]) | Converting a dsv (delimiter-separated values) string to the list of it's components. |
| [`mk_obj_of_str`](_autosummary/py2store.key_mappers.tuples.html.md#py2store.key_mappers.tuples.mk_obj_of_str)(constructor) | Make a function that transforms a string to an object. |
| [`mk_str_of_obj`](_autosummary/py2store.key_mappers.tuples.html.md#py2store.key_mappers.tuples.mk_str_of_obj)(attrs) | Make a function that transforms objects to strings, using specific attributes of object. |
| `str_of_dict`(d, str_format) | |
| [`str_of_tuple`](_autosummary/py2store.key_mappers.tuples.html.md#py2store.key_mappers.tuples.str_of_tuple)(d, str_format) | Convert tuple to str. |
| `tuple_of_dict`(d, fields) | |
| `tuple_of_str`(d, compiled_regex) | |
### py2store.key_mappers.tuples.dsv_of_list(d, sep=',')
Converting a list of strings to a dsv (delimiter-separated values) string.
Note that unlike most key mappers, there is no schema imposing size here. If you wish to impose a size
validation, do so externally (we suggest using a decorator for that).
* **Parameters:**
* **d** – A list of component strings
* **sep** – The delimiter text used to separate a string into a list of component strings
* **Returns:**
The delimiter-separated values (dsv) string for the input tuple
```pycon
>>> dsv_of_list(['a', 'brown', 'fox'], sep=' ')
'a brown fox'
>>> dsv_of_list(('jumps', 'over'), sep='/') # for filepaths (and see that tuple inputs work too!)
'jumps/over'
>>> dsv_of_list(['Sat', 'Jan', '1', '1983'], sep=',') # csv: the usual delimiter-separated values format
'Sat,Jan,1,1983'
>>> dsv_of_list(['First', 'Last'], sep=':::') # a longer delimiter
'First:::Last'
>>> dsv_of_list(['singleton'], sep='@') # when the list has only one element
'singleton'
>>> dsv_of_list([], sep='@') # when the list is empty
''
```
### py2store.key_mappers.tuples.list_of_dsv(d, sep=',')
Converting a dsv (delimiter-separated values) string to the list of it’s components.
* **Parameters:**
* **d** – A (delimiter-separated values) string
* **sep** – The delimiter text used to separate the string into a list of component strings
* **Returns:**
A list of component strings corresponding to the input delimiter-separated values (dsv) string
```pycon
>>> list_of_dsv('a brown fox', sep=' ')
['a', 'brown', 'fox']
>>> tuple(list_of_dsv('jumps/over', sep='/')) # for filepaths
('jumps', 'over')
>>> list_of_dsv('Sat,Jan,1,1983', sep=',') # csv: the usual delimiter-separated values format
['Sat', 'Jan', '1', '1983']
>>> list_of_dsv('First:::Last', sep=':::') # a longer delimiter
['First', 'Last']
>>> list_of_dsv('singleton', sep='@') # when the list has only one element
['singleton']
>>> list_of_dsv('', sep='@') # when the string is empty
[]
```
### py2store.key_mappers.tuples.mk_obj_of_str(constructor)
Make a function that transforms a string to an object. The factory making inverses of what mk_str_from_obj makes.
* **Parameters:**
**constructor** – The function (or class) that will be used to make objects from the `**kwargs` parsed out of the
string.
* **Returns:**
A function factory.
### py2store.key_mappers.tuples.mk_str_of_obj(attrs)
Make a function that transforms objects to strings, using specific attributes of object.
* **Parameters:**
**attrs** – Attributes that should be read off of the object to make the parameters of the string
* **Returns:**
A transformation function
```pycon
>>> from dataclasses import dataclass
>>> @dataclass
... class A:
... foo: int
... bar: str
>>> a = A(foo=0, bar='rin')
>>> a
A(foo=0, bar='rin')
>>>
>>> str_from_obj = mk_str_of_obj(['foo', 'bar'])
>>> str_from_obj(a, 'ST{foo}/{bar}/G')
'ST0/rin/G'
```
### py2store.key_mappers.tuples.str_of_tuple(d, str_format)
Convert tuple to str.
It’s just `str_format.format(*d)`. Why even write such a function?
1. To have a consistent interface for key conversions
2. We want a KeyValidationError to occur here
* **Parameters:**
* **d** – tuple if params to str_format
* **str_format** – Auto fields format string. If you have manual fields, consider auto_field_format_str to convert.
* **Returns:**
parametrized string
```pycon
>>> str_of_tuple(('hello', 'world'), "Well, {} dear {}!")
'Well, hello dear world!'
```
# _autosummary/py2store.misc.html.md
# py2store.misc
Functions to read from and write to misc sources.
Forwards to dol.misc
# _autosummary/py2store.mixins.html.md
# py2store.mixins
Forwards to dol.mixins:
Mixins
# _autosummary/py2store.my.grabbers.html.md
# py2store.my.grabbers
Grabbers: fetch an object from a key (a path or URL) and post-process it by kind.
A grabber is `py2store.misc.get_obj` with optional key and value transformations. The
`'ipython'` grabber turns image, WAV audio and HTML bytes into the matching IPython display
objects, so that grabbing a file in a notebook shows it.
Main entry points:
- `mk_grabber`: build a grabber from `key_trans` and `val_trans` functions
- `grabber_for`: the ready-made grabbers by name (`'ipython'`)
- `ipython_display_val_trans`: the value transformation behind the `'ipython'` grabber
### Functions
| [`fullpath`](_autosummary/py2store.my.grabbers.html.md#py2store.my.grabbers.fullpath)(path) | The absolute path of `path`, with a leading `~` expanded. |
|-----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`grabber_for`](_autosummary/py2store.my.grabbers.html.md#py2store.my.grabbers.grabber_for)(kind) | The ready-made grabber named `kind`: `'ipython'` gives `mk_grabber(val_trans=ipython_display_val_trans)`. |
| [`ipython_display_val_trans`](_autosummary/py2store.my.grabbers.html.md#py2store.my.grabbers.ipython_display_val_trans)(val[, key]) | Wrap `val` (bytes) in an IPython display object by content: `Image` for image data, `Audio` for WAV data, `HTML` for HTML (by the `key`'s extension when `key` is a string longer than 4 characters, else by a `` start); anything else is returned unchanged. |
| [`mk_grabber`](_autosummary/py2store.my.grabbers.html.md#py2store.my.grabbers.mk_grabber)(\*[, key_trans, val_trans]) | Make a function that fetches an object with `get_obj`, with optional pre- and post-processing. |
### py2store.my.grabbers.fullpath(path)
The absolute path of `path`, with a leading `~` expanded.
### py2store.my.grabbers.grabber_for(kind)
The ready-made grabber named `kind`: `'ipython'` gives `mk_grabber(val_trans=ipython_display_val_trans)`.
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If `kind` is not a known grabber name.
### py2store.my.grabbers.ipython_display_val_trans(val, key=None)
Wrap `val` (bytes) in an IPython display object by content: `Image` for image data, `Audio` for WAV data, `HTML` for HTML (by the `key`’s extension when `key` is a string longer than 4 characters, else by a `` start); anything else is returned unchanged.
Requires IPython, and uses the `imghdr` module to detect images.
### py2store.my.grabbers.mk_grabber(, key_trans=None, val_trans=None)
Make a function that fetches an object with `get_obj`, with optional pre- and post-processing.
* **Parameters:**
* **key_trans** – Applied to the key before fetching (to strip whitespace or expand a path, say).
* **val_trans** – Applied as `val_trans(value, key)` to the fetched object before it is returned.
* **Returns:**
A function `grab(k, *args, **kwargs)` that forwards `*args` and `**kwargs` to `get_obj`.
```pycon
>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), 'hello.txt')
>>> _ = open(path, 'w').write('world')
>>> grab = mk_grabber(key_trans=str.strip, val_trans=lambda v, k: v.upper())
>>> grab(' ' + path + ' ')
'WORLD'
```
# _autosummary/py2store.my.html.md
# py2store.my
functionalities meant to be configurable
### Modules
| [`grabbers`](_autosummary/py2store.my.grabbers.html.md#module-py2store.my.grabbers) | Grabbers: fetch an object from a key (a path or URL) and post-process it by kind. |
|-----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|
# _autosummary/py2store.naming.html.md
# py2store.naming
Forwards to dol.naming:
This module is about generating, validating, and operating on (parametrized) fields (i.e. stings, e.g. paths).
# _autosummary/py2store.parse_format.html.md
# py2store.parse_format
Modified from [https://github.com/r1chardj0n3s/parse](https://github.com/r1chardj0n3s/parse)
Parse strings using a specification based on the Python format() syntax.
> `parse()` is the opposite of `format()`
From there it’s a simple thing to parse a string:
```pycon
>>> parse("It's {}, I love it!", "It's spam, I love it!")
>>> _[0]
'spam'
```
Or to search a string for some pattern:
```pycon
>>> search('Age: {:d}\n', 'Name: Rufus\nAge: 42\nColor: red\n')
```
Or find all the occurrences of some pattern in a string:
```pycon
>>> ''.join(r.fixed[0] for r in findall(">{}<", "the bold text
"))
'the bold text'
```
If you’re going to use the same pattern to match lots of strings you can
compile it once:
```pycon
>>> p = compile("It's {}, I love it!")
>>> print(p)
>>> p.parse("It's spam, I love it!")
```
(“compile” is not exported for `import *` usage as it would override the
built-in `compile()` function)
The default behaviour is to match strings case insensitively. You may match with
case by specifying `case_sensitive=True`:
```pycon
>>> parse('SPAM', 'spam', case_sensitive=True) is None
True
```
## Format Syntax
A basic version of the
```
`Format String Syntax`_
```
is supported with anonymous
(fixed-position), named and formatted fields:
```default
{[field name]:[format spec]}
```
Field names must be a valid Python identifiers, including dotted names;
element indexes imply dictionaries (see below for example).
Numbered fields are also not supported: the result of parsing will include
the parsed fields in the order they are parsed.
The conversion of fields to types other than strings is done based on the
type in the format specification, which mirrors the `format()` behaviour.
There are no “!” field conversions like `format()` has.
Some simple parse() format string examples:
```pycon
>>> parse("Bring me a {}", "Bring me a shrubbery")
>>> r = parse("The {} who say {}", "The knights who say Ni!")
>>> print(r)
>>> print(r.fixed)
('knights', 'Ni!')
>>> r = parse("Bring out the holy {item}", "Bring out the holy hand grenade")
>>> print(r)
>>> print(r.named)
{'item': 'hand grenade'}
>>> print(r['item'])
hand grenade
```
Dotted names and indexes are possible though the application must make
additional sense of the result:
```pycon
>>> r = parse("Mmm, {food.type}, I love it!", "Mmm, spam, I love it!")
>>> print(r)
>>> print(r.named)
{'food.type': 'spam'}
>>> print(r['food.type'])
spam
>>> r = parse("My quest is {quest[name]}", "My quest is to seek the holy grail!")
>>> print(r)
>>> print(r['quest'])
{'name': 'to seek the holy grail!'}
>>> print(r['quest']['name'])
to seek the holy grail!
```
If the text you’re matching has braces in it you can match those by including
a double-brace `{{` or `}}` in your format string, just like format() does.
## Format Specification
Most often a straight format-less `{}` will suffice where a more complex
format specification might have been used.
Most of `format()`’s
```
`Format Specification Mini-Language`_
```
is supported:
> [[fill]align][0][width][.precision][type]
The differences between `parse()` and `format()` are:
- The align operators will cause spaces (or specified fill character) to be
stripped from the parsed value. The width is not enforced; it just indicates
there may be whitespace or “0”s to strip.
- Numeric parsing will automatically handle a “0b”, “0o” or “0x” prefix.
That is, the “#” format character is handled automatically by d, b, o
and x formats. For “d” any will be accepted, but for the others the correct
prefix must be present if at all.
- Numeric sign is handled automatically.
- The thousands separator is handled automatically if the “n” type is used.
- The types supported are a slightly different mix to the format() types. Some
format() types come directly over: “d”, “n”, “%”, “f”, “e”, “b”, “o” and “x”.
In addition some regular expression character group types “D”, “w”, “W”, “s”
and “S” are also available.
- The “e” and “g” types are case-insensitive so there is not need for
the “E” or “G” types.
| Type | Characters Matched | Output |
|--------|------------------------------------------------------------------------------------|----------|
| w | Letters and underscore | str |
| W | Non-letter and underscore | str |
| s | Whitespace | str |
| S | Non-whitespace | str |
| d | Digits (effectively integer numbers) | int |
| D | Non-digit | str |
| n | Numbers with thousands separators (, or .) | int |
| % | Percentage (converted to value/100.0) | float |
| f | Fixed-point numbers | float |
| F | Decimal numbers | Decimal |
| e | Floating-point numbers with exponent
e.g. 1.1e-10, NAN (all case insensitive) | float |
| g | General number format (either d, f or e) | float |
| b | Binary numbers | int |
| o | Octal numbers | int |
| x | Hexadecimal numbers (lower and upper case) | int |
| ti | ISO 8601 format date/time
e.g. 1972-01-20T10:21:36Z (“T” and “Z”
optional) | datetime |
| te | RFC2822 e-mail format date/time
e.g. Mon, 20 Jan 1972 10:21:36 +1000 | datetime |
| tg | Global (day/month) format date/time
e.g. 20/1/1972 10:21:36 AM +1:00 | datetime |
| ta | US (month/day) format date/time
e.g. 1/20/1972 10:21:36 PM +10:30 | datetime |
| tc | ctime() format date/time
e.g. Sun Sep 16 01:03:52 1973 | datetime |
| th | HTTP log format date/time
e.g. 21/Nov/2011:00:07:11 +0000 | datetime |
| ts | Linux system log format date/time
e.g. Nov 9 03:37:44 | datetime |
| tt | Time
e.g. 10:21:36 PM -5:30 | time |
Some examples of typed parsing with `None` returned if the typing
does not match:
```pycon
>>> parse('Our {:d} {:w} are...', 'Our 3 weapons are...')
>>> parse('Our {:d} {:w} are...', 'Our three weapons are...')
>>> parse('Meet at {:tg}', 'Meet at 1/2/2011 11:00 PM')
```
And messing about with alignment:
```pycon
>>> parse('with {:>} herring', 'with a herring')
>>> parse('spam {:^} spam', 'spam lovely spam')
```
Note that the “center” alignment does not test to make sure the value is
centered - it just strips leading and trailing whitespace.
Width and precision may be used to restrict the size of matched text
from the input. Width specifies a minimum size and precision specifies
a maximum. For example:
```pycon
>>> parse('{:.2}{:.2}', 'look') # specifying precision
>>> parse('{:4}{:4}', 'look at that') # specifying width
>>> parse('{:4}{:.4}', 'look at that') # specifying both
>>> parse('{:2d}{:2d}', '0440') # parsing two contiguous numbers
```
Some notes for the date and time types:
- the presence of the time part is optional (including ISO 8601, starting
at the “T”). A full datetime object will always be returned; the time
will be set to 00:00:00. You may also specify a time without seconds.
- when a seconds amount is present in the input fractions will be parsed
to give microseconds.
- except in ISO 8601 the day and month digits may be 0-padded.
- the date separator for the tg and ta formats may be “-” or “/”.
- named months (abbreviations or full names) may be used in the ta and tg
formats in place of numeric months.
- as per RFC 2822 the e-mail format may omit the day (and comma), and the
seconds but nothing else.
- hours greater than 12 will be happily accepted.
- the AM/PM are optional, and if PM is found then 12 hours will be added
to the datetime object’s hours amount - even if the hour is greater
than 12 (for consistency.)
- in ISO 8601 the “Z” (UTC) timezone part may be a numeric offset
- timezones are specified as “+HH:MM” or “-HH:MM”. The hour may be one or two
digits (0-padded is OK.) Also, the “:” is optional.
- the timezone is optional in all except the e-mail format (it defaults to
UTC.)
- named timezones are not handled yet.
#### NOTE
attempting to match too many datetime fields in a single parse() will
currently result in a resource allocation issue. A TooManyFields exception
will be raised in this instance. The current limit is about 15. It is hoped
that this limit will be removed one day.
## Result and Match Objects
The result of a `parse()` and `search()` operation is either `None` (no match), a
`Result` instance or a `Match` instance if `evaluate_result` is False.
The `Result` instance has three attributes:
fixed
: A tuple of the fixed-position, anonymous fields extracted from the input.
named
: A dictionary of the named fields extracted from the input.
spans
: A dictionary mapping the names and fixed position indices matched to a
2-tuple slice range of where the match occurred in the input.
The span does not include any stripped padding (alignment or width).
The `Match` instance has one method:
evaluate_result()
: Generates and returns a `Result` instance for this `Match` object.
## Custom Type Conversions
If you wish to have matched fields automatically converted to your own type you
may pass in a dictionary of type conversion information to `parse()` and
`compile()`.
The converter will be passed the field string matched. Whatever it returns
will be substituted in the `Result` instance for that field.
Your custom type conversions may override the builtin types if you supply one
with the same identifier.
```pycon
>>> def shouty(string):
... return string.upper()
...
>>> parse('{:shouty} world', 'hello world', dict(shouty=shouty))
```
If the type converter has the optional `pattern` attribute, it is used as
regular expression for better pattern matching (instead of the default one).
```pycon
>>> def parse_number(text):
... return int(text)
>>> parse_number.pattern = r'\d+'
>>> parse('Answer: {number:Number}', 'Answer: 42', dict(Number=parse_number))
>>> _ = parse('Answer: {:Number}', 'Answer: Alice', dict(Number=parse_number))
>>> assert _ is None, "MISMATCH"
```
You can also use the `with_pattern(pattern)` decorator to add this
information to a type converter function:
```pycon
>>> @with_pattern(r'\d+')
... def parse_number(text):
... return int(text)
>>> parse('Answer: {number:Number}', 'Answer: 42', dict(Number=parse_number))
```
A more complete example of a custom type might be:
```pycon
>>> yesno_mapping = {
... "yes": True, "no": False,
... "on": True, "off": False,
... "true": True, "false": False,
... }
>>> @with_pattern(r"|".join(yesno_mapping))
... def parse_yesno(text):
... return yesno_mapping[text.lower()]
```
If the type converter `pattern` uses regex-grouping (with parenthesis),
you should indicate this by using the optional `regex_group_count` parameter
in the `with_pattern()` decorator:
```pycon
>>> @with_pattern(r'((\d+))', regex_group_count=2)
... def parse_number2(text):
... return int(text)
>>> parse('Answer: {:Number2} {:Number2}', 'Answer: 42 43', dict(Number2=parse_number2))
```
Otherwise, this may cause parsing problems with unnamed/fixed parameters.
## Potential Gotchas
`parse()` will always match the shortest text necessary (from left to right)
to fulfil the parse pattern, so for example:
```pycon
>>> pattern = '{dir1}/{dir2}'
>>> data = 'root/parent/subdir'
>>> sorted(parse(pattern, data).named.items())
[('dir1', 'root'), ('dir2', 'parent/subdir')]
```
So, even though `{'dir1': 'root/parent', 'dir2': 'subdir'}` would also fit
the pattern, the actual match represents the shortest successful match for
`dir1`.
---
**Version history (in brief)**:
- 1.9.0 We now honor precision and width specifiers when parsing numbers
and strings, allowing parsing of concatenated elements of fixed width
(thanks Julia Signell)
- 1.8.4 Add LICENSE file at request of packagers.
Correct handling of AM/PM to follow most common interpretation.
Correct parsing of hexadecimal that looks like a binary prefix.
Add ability to parse case sensitively.
Add parsing of numbers to Decimal with “F” (thanks John Vandenberg)
- 1.8.3 Add regex_group_count to with_pattern() decorator to support
user-defined types that contain brackets/parenthesis (thanks Jens Engel)
- 1.8.2 add documentation for including braces in format string
- 1.8.1 ensure bare hexadecimal digits are not matched
- 1.8.0 support manual control over result evaluation (thanks Timo Furrer)
- 1.7.0 parse dict fields (thanks Mark Visser) and adapted to allow
more than 100 re groups in Python 3.5+ (thanks David King)
- 1.6.6 parse Linux system log dates (thanks Alex Cowan)
- 1.6.5 handle precision in float format (thanks Levi Kilcher)
- 1.6.4 handle pipe “|” characters in parse string (thanks Martijn Pieters)
- 1.6.3 handle repeated instances of named fields, fix bug in PM time
overflow
- 1.6.2 fix logging to use local, not root logger (thanks Necku)
- 1.6.1 be more flexible regarding matched ISO datetimes and timezones in
general, fix bug in timezones without “:” and improve docs
- 1.6.0 add support for optional `pattern` attribute in user-defined types
(thanks Jens Engel)
- 1.5.3 fix handling of question marks
- 1.5.2 fix type conversion error with dotted names (thanks Sebastian Thiel)
- 1.5.1 implement handling of named datetime fields
- 1.5 add handling of dotted field names (thanks Sebastian Thiel)
- 1.4.1 fix parsing of “0” in int conversion (thanks James Rowe)
- 1.4 add \_\_getitem_\_ convenience access on Result.
- 1.3.3 fix Python 2.5 setup.py issue.
- 1.3.2 fix Python 3.2 setup.py issue.
- 1.3.1 fix a couple of Python 3.2 compatibility issues.
- 1.3 added search() and findall(); removed compile() from `import *`
export as it overwrites builtin.
- 1.2 added ability for custom and override type conversions to be
provided; some cleanup
- 1.1.9 to keep things simpler number sign is handled automatically;
significant robustification in the face of edge-case input.
- 1.1.8 allow “d” fields to have number base “0x” etc. prefixes;
fix up some field type interactions after stress-testing the parser;
implement “%” type.
- 1.1.7 Python 3 compatibility tweaks (2.5 to 2.7 and 3.2 are supported).
- 1.1.6 add “e” and “g” field types; removed redundant “h” and “X”;
removed need for explicit “#”.
- 1.1.5 accept textual dates in more places; Result now holds match span
positions.
- 1.1.4 fixes to some int type conversion; implemented “=” alignment; added
date/time parsing with a variety of formats handled.
- 1.1.3 type conversion is automatic based on specified field types. Also added
“f” and “n” types.
- 1.1.2 refactored, added compile() and limited `from parse import *`
- 1.1.1 documentation improvements
- 1.1.0 implemented more of the
```
`Format Specification Mini-Language`_
```
and removed the restriction on mixing fixed-position and named fields
- 1.0.0 initial release
This code is copyright 2012-2017 Richard Jones <[richard@python.org](mailto:richard@python.org)>
See the end of the source file for the license of use.
### Functions
| [`parse`](_autosummary/py2store.parse_format.html.md#py2store.parse_format.parse)(format, string[, extra_types, ...]) | Using "format" attempt to pull values from "string". |
|----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|
| [`search`](_autosummary/py2store.parse_format.html.md#py2store.parse_format.search)(format, string[, pos, endpos, ...]) | Search "string" for the first occurrence of "format". |
| [`findall`](_autosummary/py2store.parse_format.html.md#py2store.parse_format.findall)(format, string[, pos, endpos, ...]) | Search "string" for all occurrences of "format". |
| [`with_pattern`](_autosummary/py2store.parse_format.html.md#py2store.parse_format.with_pattern)(pattern[, regex_group_count]) | Attach a regular expression pattern matcher to a custom type converter function. |
### py2store.parse_format.findall(format, string, pos=0, endpos=None, extra_types=None, evaluate_result=True, case_sensitive=False)
Search “string” for all occurrences of “format”.
You will be returned an iterator that holds Result instances
for each format match found.
Optionally start the search at “pos” character index and limit the search
to a maximum index of endpos - equivalent to search(string[:endpos]).
If `evaluate_result` is True each returned Result instance has two attributes:
> .fixed - tuple of fixed-position values from the string
> .named - dict of named values from the string
If `evaluate_result` is False each returned value is a Match instance with one method:
> .evaluate_result() - This will return a Result instance like you would get with `evaluate_result` set to True
The default behaviour is to match strings case insensitively. You may match with
case by specifying case_sensitive=True.
If the format is invalid a ValueError will be raised.
See the module documentation for the use of “extra_types”.
### py2store.parse_format.parse(format, string, extra_types=None, evaluate_result=True, case_sensitive=False)
Using “format” attempt to pull values from “string”.
The format must match the string contents exactly. If the value
you’re looking for is instead just a part of the string use
search().
If `evaluate_result` is True the return value will be an Result instance with two attributes:
> .fixed - tuple of fixed-position values from the string
> .named - dict of named values from the string
If `evaluate_result` is False the return value will be a Match instance with one method:
> .evaluate_result() - This will return a Result instance like you would get with `evaluate_result` set to True
The default behaviour is to match strings case insensitively. You may match with
case by specifying case_sensitive=True.
If the format is invalid a ValueError will be raised.
See the module documentation for the use of “extra_types”.
In the case there is no match parse() will return None.
### py2store.parse_format.search(format, string, pos=0, endpos=None, extra_types=None, evaluate_result=True, case_sensitive=False)
Search “string” for the first occurrence of “format”.
The format may occur anywhere within the string. If
instead you wish for the format to exactly match the string
use parse().
Optionally start the search at “pos” character index and limit the search
to a maximum index of endpos - equivalent to search(string[:endpos]).
If `evaluate_result` is True the return value will be an Result instance with two attributes:
> .fixed - tuple of fixed-position values from the string
> .named - dict of named values from the string
If `evaluate_result` is False the return value will be a Match instance with one method:
> .evaluate_result() - This will return a Result instance like you would get with `evaluate_result` set to True
The default behaviour is to match strings case insensitively. You may match with
case by specifying case_sensitive=True.
If the format is invalid a ValueError will be raised.
See the module documentation for the use of “extra_types”.
In the case there is no match parse() will return None.
### py2store.parse_format.with_pattern(pattern, regex_group_count=None)
Attach a regular expression pattern matcher to a custom type converter
function.
This annotates the type converter with the `pattern` attribute.
### Example
```pycon
>>> @with_pattern(r"\d+")
... def parse_number(text):
... return int(text)
```
is equivalent to:
```pycon
>>> def parse_number(text):
... return int(text)
>>> parse_number.pattern = r"\d+"
```
* **Parameters:**
* **pattern** – regular expression pattern (as text)
* **regex_group_count** – Indicates how many regex-groups are in pattern.
* **Returns:**
wrapped function
# _autosummary/py2store.paths.html.md
# py2store.paths
Forwards to dol.paths:
Module for path (and path-like) object manipulation
# _autosummary/py2store.persisters.dropbox_w_dropbox.html.md
# py2store.persisters.dropbox_w_dropbox
Forwards to dropboxdol
# _autosummary/py2store.persisters.googledrive_w_pydrive.html.md
# py2store.persisters.googledrive_w_pydrive
Forwards to pydrivedol
# _autosummary/py2store.persisters.html.md
# py2store.persisters
base persisters – now all forwarding to separate libraries
### Modules
| [`dropbox_w_dropbox`](_autosummary/py2store.persisters.dropbox_w_dropbox.html.md#module-py2store.persisters.dropbox_w_dropbox) | Forwards to dropboxdol |
|-------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|
| [`googledrive_w_pydrive`](_autosummary/py2store.persisters.googledrive_w_pydrive.html.md#module-py2store.persisters.googledrive_w_pydrive) | Forwards to pydrivedol |
| [`local_files`](_autosummary/py2store.persisters.local_files.html.md#module-py2store.persisters.local_files) | Base classes and helpers to read and write local files as key-value collections. |
| [`new_s3`](_autosummary/py2store.persisters.new_s3.html.md#module-py2store.persisters.new_s3) | Forwards to s3dol.new_s3 |
| [`redis_w_redis`](_autosummary/py2store.persisters.redis_w_redis.html.md#module-py2store.persisters.redis_w_redis) | Forwards to redisdol |
| [`s3_w_boto3`](_autosummary/py2store.persisters.s3_w_boto3.html.md#module-py2store.persisters.s3_w_boto3) | Forwards to s3dol.s3_w_boto3 |
| [`sql_w_sqlalchemy`](_autosummary/py2store.persisters.sql_w_sqlalchemy.html.md#module-py2store.persisters.sql_w_sqlalchemy) | Forwards to sqldol |
| [`w_aiofile`](_autosummary/py2store.persisters.w_aiofile.html.md#module-py2store.persisters.w_aiofile) | Forwards to aiofiledol |
# _autosummary/py2store.persisters.local_files.html.md
# py2store.persisters.local_files
Base classes and helpers to read and write local files as key-value collections.
Keys are full file paths. The pieces here (path listing, key validation from a path template,
read, write and delete through `open`) are what `py2store.stores.local_store` assembles
into stores with relative keys.
Main entry points:
- `FileReader`: a directory as a read-only mapping (subdirectories give nested readers, files give `bytes`)
- `DirReader`: the subdirectories of a directory, as a mapping
- `PathFormatPersister`: read, write and delete the files whose paths match a template
- `iter_filepaths_in_folder_recursively`: the paths of all files under a folder
```pycon
>>> import os, tempfile
>>> rootdir = tempfile.mkdtemp()
>>> _ = open(os.path.join(rootdir, 'a.txt'), 'w').write('hi')
>>> [os.path.basename(p) for p in iter_filepaths_in_folder_recursively(rootdir)]
['a.txt']
```
### Functions
| [`dirpaths_in_dir`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.dirpaths_in_dir)(rootdir) | The full paths of the folders directly under `rootdir`. |
|----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`endswith_slash`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.endswith_slash)(path) | Whether `path` ends with the OS path separator. |
| [`ensure_slash_suffix`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.ensure_slash_suffix)(path) | Add a file separation (/ or ) at the end of path str, if not already present. |
| [`extend_prefix`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.extend_prefix)(prefix, new_prefix) | Join `new_prefix` to `prefix`, with a trailing separator. |
| [`filepaths_in_dir`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.filepaths_in_dir)(rootdir) | The full paths of the files (not the folders) directly under `rootdir`. |
| [`first_non_existing_parent_dir`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.first_non_existing_parent_dir)(dirpath) | The highest ancestor directory of `dirpath` that does not exist, or `''` if they all exist. |
| [`iter_dirpaths_in_folder_recursively`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.iter_dirpaths_in_folder_recursively)(root_folder) | Yield the full paths of the folders under `root_folder`, recursively (`max_levels` as in `iter_filepaths_in_folder_recursively`). |
| [`iter_filepaths_in_folder`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.iter_filepaths_in_folder)(root_folder) | The full paths of the files and folders directly under `root_folder`. |
| [`iter_filepaths_in_folder_recursively`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.iter_filepaths_in_folder_recursively)(root_folder) | Yield the full paths of the files under `root_folder`, recursively. |
| [`iter_relative_files_and_folder`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.iter_relative_files_and_folder)(root_folder) | The names of the files and folders directly under `root_folder`. |
| [`path_match_regex_from_path_format`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.path_match_regex_from_path_format)(path_format) | Compile the regex that full paths matching the `path_format` template satisfy (a bare directory matches everything under it). |
| [`paths_in_dir`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.paths_in_dir)(rootdir) | The full paths of the files and folders directly under `rootdir`. |
| [`paths_in_dir_with_slash_suffix_for_dirs`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.paths_in_dir_with_slash_suffix_for_dirs)(rootdir) | Yield the full paths directly under `rootdir`, with a trailing separator on directories. |
| [`pattern_filter`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.pattern_filter)(pattern) | Make a predicate that is true for strings matching the regex `pattern` from their start. |
| [`w_helpful_folder_not_found_error`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.w_helpful_folder_not_found_error)(\*[, ...]) | Decorator factory: re-raise `caught_errors` from a method as `raise_error`, with the original message plus `extra_msg` (a string, or a callable of the method's arguments). |
### Classes
| [`DirReader`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.DirReader)(rootdir) | KV Reader whose keys are the full paths of the subdirectories of `rootdir` and whose values are `DirReader` instances of them. |
|------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|
| [`DirpathFormatKeys`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.DirpathFormatKeys)(path_format[, max_levels]) | Keys collection of the folders matching a path template, recursively under its root (`max_levels` limits the depth). |
| [`FileReader`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.FileReader)(rootdir) | KV Reader whose keys are paths and values are: |
| [`FilepathFormatKeys`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.FilepathFormatKeys)(path_format[, max_levels]) | Keys collection of the files matching a path template, recursively under its root (`max_levels` limits the depth). |
| [`LocalFileRWD`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.LocalFileRWD)([mode]) | A class providing get, set and delete functionality using local files as the storage backend. |
| [`LocalFileStreamGetter`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.LocalFileStreamGetter)(\*\*open_kwargs) | A class to get stream objects of local open files. |
| [`PathFormat`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormat)(path_format) | Key validation from a path template: which full paths belong to the collection. |
| [`PathFormatPersister`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormatPersister)(path_format[, ...]) | Read, write and delete local files whose full paths match a path template. |
| [`PrefixedDirpathsRecursive`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PrefixedDirpathsRecursive)() | Keys collection for local files, where the keys are full filepaths RECURSIVELY under a given root dir \_prefix. |
| [`PrefixedFilepaths`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PrefixedFilepaths)() | Keys collection for local files, where the keys are full filepaths DIRECTLY under a given root dir \_prefix. |
| [`PrefixedFilepathsRecursive`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PrefixedFilepathsRecursive)() | Keys collection for local files, where the keys are full filepaths RECURSIVELY under a given root dir \_prefix. |
### Exceptions
| [`FolderNotFoundError`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.FolderNotFoundError) | Raised when writing to a path whose directory does not exist; the message names the first missing directory. |
|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|
### *class* py2store.persisters.local_files.DirReader(rootdir)
Bases: [`FileReader`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.FileReader)
KV Reader whose keys are the full paths of the subdirectories of `rootdir` and whose values are `DirReader` instances of them.
```pycon
>>> import os, tempfile
>>> rootdir = tempfile.mkdtemp()
>>> _ = open(os.path.join(rootdir, 'a.txt'), 'wb').write(b'hi')
>>> os.mkdir(os.path.join(rootdir, 'sub'))
>>> s = DirReader(rootdir)
>>> [k[len(s.rootdir):] for k in s]
['sub/']
>>> os.path.join(rootdir, 'a.txt') in s
False
>>> type(s[os.path.join(rootdir, 'sub', '')]).__name__
'DirReader'
```
### *class* py2store.persisters.local_files.DirpathFormatKeys(path_format, max_levels=inf)
Bases: [`PathFormat`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormat), `FilteredKeysMixin`, `KeyValidationABC`, [`PrefixedDirpathsRecursive`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PrefixedDirpathsRecursive), `IterBasedSizedMixin`
Keys collection of the folders matching a path template, recursively under its root (`max_levels` limits the depth).
### *class* py2store.persisters.local_files.FileReader(rootdir)
Bases: `KvReader`
KV Reader whose keys are paths and values are:
- Another FileReader if a path points to a directory
- The bytes of the file if the path points to a file.
Keys are the full paths directly under `rootdir`; directory keys end with a separator.
```pycon
>>> import os, tempfile
>>> rootdir = tempfile.mkdtemp()
>>> _ = open(os.path.join(rootdir, 'a.txt'), 'wb').write(b'hi')
>>> os.mkdir(os.path.join(rootdir, 'sub'))
>>> s = FileReader(rootdir)
>>> sorted(k[len(s.rootdir):] for k in s)
['a.txt', 'sub/']
>>> s[os.path.join(rootdir, 'a.txt')]
b'hi'
>>> type(s[os.path.join(rootdir, 'sub', '')]).__name__
'FileReader'
```
### *class* py2store.persisters.local_files.FilepathFormatKeys(path_format, max_levels=inf)
Bases: [`PathFormat`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormat), `FilteredKeysMixin`, `KeyValidationABC`, [`PrefixedFilepathsRecursive`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PrefixedFilepathsRecursive), `IterBasedSizedMixin`
Keys collection of the files matching a path template, recursively under its root (`max_levels` limits the depth).
### *exception* py2store.persisters.local_files.FolderNotFoundError
Bases: `NoSuchKeyError`
Raised when writing to a path whose directory does not exist; the message names the first missing directory.
### *class* py2store.persisters.local_files.LocalFileRWD(mode='', \*\*open_kwargs)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A class providing get, set and delete functionality using local files as the storage backend.
### *class* py2store.persisters.local_files.LocalFileStreamGetter(\*\*open_kwargs)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A class to get stream objects of local open files.
The class can only get keys, and only to read, write (destructive or append).
```pycon
>>> from tempfile import mkdtemp
>>> import os
>>> rootdir = mkdtemp()
>>>
>>> appendable_stream = LocalFileStreamGetter(mode='a+')
>>> reader = PathFormatPersister(rootdir)
>>> filepath = os.path.join(rootdir, 'tmp.txt')
>>>
>>> with appendable_stream[filepath] as fp:
... fp.write('hello')
5
>>> print(reader[filepath])
hello
>>> with appendable_stream[filepath] as fp:
... fp.write(' world')
6
>>>
>>> print(reader[filepath])
hello world
```
### *class* py2store.persisters.local_files.PathFormat(path_format)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Key validation from a path template: which full paths belong to the collection.
* **Parameters:**
**path_format** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The f-string template that the full path keys should match: a root
directory (`'/data/'`, everything under it) or a template such as
`'/data/{}.csv'` (only `.csv` files under `/data/`). The directory
containing the part before the first `{` is the root, available as `_prefix`.
```pycon
>>> pf = PathFormat('/data/{}.csv')
>>> pf._prefix, pf.is_valid_key('/data/a.csv'), pf.is_valid_key('/data/a.txt')
('/data/', True, False)
```
#### is_valid_key(k)
Whether `k` matches the path template.
### *class* py2store.persisters.local_files.PathFormatPersister(path_format, max_levels=inf, mode='', \*\*open_kwargs)
Bases: [`FilepathFormatKeys`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.FilepathFormatKeys), [`LocalFileRWD`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.LocalFileRWD)
Read, write and delete local files whose full paths match a path template.
* **Parameters:**
* **path_format** – The path template (see `PathFormat`).
* **max_levels** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – How many folder levels below the root to include when iterating.
* **mode** – `''`, `'t'` or `'b'`: whether files are opened in text or binary mode.
* **\*\*open_kwargs** – Forwarded to `open`; `read_mode` and `write_mode` entries override
the modes derived from `mode`.
### *class* py2store.persisters.local_files.PrefixedDirpathsRecursive
Bases: [`PrefixedFilepaths`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PrefixedFilepaths)
Keys collection for local files, where the keys are full filepaths RECURSIVELY under a given root dir \_prefix.
This mixin adds iteration (_\_iter_\_), length (_\_len_\_), and containment (_\_contains_\_(k)).
### *class* py2store.persisters.local_files.PrefixedFilepaths
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Keys collection for local files, where the keys are full filepaths DIRECTLY under a given root dir \_prefix.
This mixin adds iteration (_\_iter_\_), length (_\_len_\_), and containment (_\_contains_\_(k)).
### *class* py2store.persisters.local_files.PrefixedFilepathsRecursive
Bases: [`PrefixedFilepaths`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PrefixedFilepaths)
Keys collection for local files, where the keys are full filepaths RECURSIVELY under a given root dir \_prefix.
This mixin adds iteration (_\_iter_\_), length (_\_len_\_), and containment (_\_contains_\_(k)).
### py2store.persisters.local_files.dirpaths_in_dir(rootdir)
The full paths of the folders directly under `rootdir`.
### py2store.persisters.local_files.endswith_slash(path)
Whether `path` ends with the OS path separator.
### py2store.persisters.local_files.ensure_slash_suffix(path)
Add a file separation (/ or ) at the end of path str, if not already present.
### py2store.persisters.local_files.extend_prefix(prefix, new_prefix)
Join `new_prefix` to `prefix`, with a trailing separator.
### py2store.persisters.local_files.filepaths_in_dir(rootdir)
The full paths of the files (not the folders) directly under `rootdir`.
### py2store.persisters.local_files.first_non_existing_parent_dir(dirpath)
The highest ancestor directory of `dirpath` that does not exist, or `''` if they all exist.
### py2store.persisters.local_files.iter_dirpaths_in_folder_recursively(root_folder, max_levels=None, \_current_level=0)
Yield the full paths of the folders under `root_folder`, recursively (`max_levels` as in `iter_filepaths_in_folder_recursively`).
### py2store.persisters.local_files.iter_filepaths_in_folder(root_folder)
The full paths of the files and folders directly under `root_folder`.
### py2store.persisters.local_files.iter_filepaths_in_folder_recursively(root_folder, max_levels=None, \_current_level=0)
Yield the full paths of the files under `root_folder`, recursively.
* **Parameters:**
* **root_folder** – The folder to walk.
* **max_levels** – How many folder levels below `root_folder` to descend into: `0` yields
only the files directly under it, `None` means no limit.
### py2store.persisters.local_files.iter_relative_files_and_folder(root_folder)
The names of the files and folders directly under `root_folder`.
### py2store.persisters.local_files.path_match_regex_from_path_format(path_format)
Compile the regex that full paths matching the `path_format` template satisfy (a bare directory matches everything under it).
### py2store.persisters.local_files.paths_in_dir(rootdir)
The full paths of the files and folders directly under `rootdir`.
### py2store.persisters.local_files.paths_in_dir_with_slash_suffix_for_dirs(rootdir)
Yield the full paths directly under `rootdir`, with a trailing separator on directories.
### py2store.persisters.local_files.pattern_filter(pattern)
Make a predicate that is true for strings matching the regex `pattern` from their start.
### py2store.persisters.local_files.w_helpful_folder_not_found_error(\*, raise_error=, extra_msg='', caught_errors=)
Decorator factory: re-raise `caught_errors` from a method as `raise_error`, with the original message plus `extra_msg` (a string, or a callable of the method’s arguments).
# _autosummary/py2store.persisters.new_s3.html.md
# py2store.persisters.new_s3
Forwards to s3dol.new_s3
# _autosummary/py2store.persisters.redis_w_redis.html.md
# py2store.persisters.redis_w_redis
Forwards to redisdol
# _autosummary/py2store.persisters.s3_w_boto3.html.md
# py2store.persisters.s3_w_boto3
Forwards to s3dol.s3_w_boto3
# _autosummary/py2store.persisters.sql_w_sqlalchemy.html.md
# py2store.persisters.sql_w_sqlalchemy
Forwards to sqldol
# _autosummary/py2store.persisters.w_aiofile.html.md
# py2store.persisters.w_aiofile
Forwards to aiofiledol
# _autosummary/py2store.serializers.html.md
# py2store.serializers
a package of serializers
### Modules
| [`pickled`](_autosummary/py2store.serializers.pickled.html.md#module-py2store.serializers.pickled) | functions to pickle objects |
|------------------------------------------------------------------------------------------------|-------------------------------|
# _autosummary/py2store.serializers.pickled.html.md
# py2store.serializers.pickled
functions to pickle objects
### Functions
| [`mk_marshal_rw_funcs`](_autosummary/py2store.serializers.pickled.html.md#py2store.serializers.pickled.mk_marshal_rw_funcs)(\*\*kwargs) | Generates a reader and writer using marshal. |
|---------------------------------------------------------------------------------------------------|------------------------------------------------|
| [`mk_pickle_rw_funcs`](_autosummary/py2store.serializers.pickled.html.md#py2store.serializers.pickled.mk_pickle_rw_funcs)([fix_imports, protocol, ...]) | Generates a reader and writer using pickle. |
### py2store.serializers.pickled.mk_marshal_rw_funcs(\*\*kwargs)
Generates a reader and writer using marshal. That is, a pair of parametrized loads and dumps
```pycon
>>> read, write = mk_marshal_rw_funcs()
>>> d = {'a': 'simple', 'and': {'a': b'more', 'complex': [1, 2.2]}}
>>> serialized_d = write(d)
>>> deserialized_d = read(serialized_d)
>>> assert d == deserialized_d
```
### py2store.serializers.pickled.mk_pickle_rw_funcs(fix_imports=True, protocol=None, pickle_encoding='ASCII', pickle_errors='strict')
Generates a reader and writer using pickle. That is, a pair of parametrized loads and dumps
```pycon
>>> read, write = mk_pickle_rw_funcs()
>>> d = {'a': 'simple', 'and': {'a': b'more', 'complex': [1, 2.2, dict]}}
>>> serialized_d = write(d)
>>> deserialized_d = read(serialized_d)
>>> assert d == deserialized_d
```
# _autosummary/py2store.signatures.html.md
# py2store.signatures
Forwards to dol.signatures:
Signature calculus
# _autosummary/py2store.slib.html.md
# py2store.slib
Data Object Layers for a few standard libs.
### Modules
| [`s_configparser`](_autosummary/py2store.slib.s_configparser.html.md#module-py2store.slib.s_configparser) | Data Object Layer for configparser standard lib. |
|-------------------------------------------------------------------------------------------------------|----------------------------------------------------|
| [`s_zipfile`](_autosummary/py2store.slib.s_zipfile.html.md#module-py2store.slib.s_zipfile) | a data object layer for zipfile |
# _autosummary/py2store.slib.s_configparser.html.md
# py2store.slib.s_configparser
Data Object Layer for configparser standard lib.
# _autosummary/py2store.slib.s_zipfile.html.md
# py2store.slib.s_zipfile
a data object layer for zipfile
# _autosummary/py2store.sources.html.md
# py2store.sources
Forwards to dol.sources:
This module contains key-value views of disparate sources.
# _autosummary/py2store.stores.dropbox_store.html.md
# py2store.stores.dropbox_store
Forwards to dropboxdol
# _autosummary/py2store.stores.html.md
# py2store.stores
a package of various stores
### Modules
| [`dropbox_store`](_autosummary/py2store.stores.dropbox_store.html.md#module-py2store.stores.dropbox_store) | Forwards to dropboxdol |
|-----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|
| [`local_store`](_autosummary/py2store.stores.local_store.html.md#module-py2store.stores.local_store) | Stores that read and write local files as key-value mappings. |
| [`s3_store`](_autosummary/py2store.stores.s3_store.html.md#module-py2store.stores.s3_store) | Forwards to s3dol.s3_store |
| [`sql_w_sqlalchemy`](_autosummary/py2store.stores.sql_w_sqlalchemy.html.md#module-py2store.stores.sql_w_sqlalchemy) | Forwards to sqldol |
# _autosummary/py2store.stores.local_store.html.md
# py2store.stores.local_store
Stores that read and write local files as key-value mappings.
Keys are paths relative to a root directory and values are the file contents (text, bytes,
or objects through pickle or json serialization). The `Local*Store` classes need the
directories to exist already; the `Quick*Store` classes create missing directories on write
and pick a temporary root when none is given.
Main entry points:
- `LocalTextStore`, `LocalBinaryStore`: file contents as `str` or `bytes`
- `LocalPickleStore`, `LocalJsonStore`: values serialized with pickle or json
- `QuickStore`: `LocalPickleStore` with a temporary default root and directories created on write
- `DirStore`: the subdirectories of a directory, as nested stores
```pycon
>>> import tempfile
>>> s = LocalTextStore(tempfile.mkdtemp())
>>> s['hello.txt'] = 'world'
>>> list(s), s['hello.txt']
(['hello.txt'], 'world')
```
### Functions
| [`mk_absolute_path`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.mk_absolute_path)(path_format) | Expand a leading `~` and make a path starting with `.` absolute; other paths are returned unchanged. |
|----------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| [`mk_tmp_quick_store_dirpath`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.mk_tmp_quick_store_dirpath)([dirname]) | Path of `dirname` under the system temp directory (`tempfile.gettempdir()`). |
### Classes
| [`AutoMkDirsOnSetitemMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.AutoMkDirsOnSetitemMixin)() | A mixin that will automatically create directories on setitem, when missing. |
|---------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| [`AutoMkPathformatMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.AutoMkPathformatMixin)([path_format, max_levels]) | A mixin that will choose a path_format if none given |
| [`DirStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.DirStore)(rootdir) | A store for local directories. |
| [`LocalBinaryStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalBinaryStore)(path_format[, max_levels]) | Local files store for binary data: like `LocalTextStore`, but values are `bytes`. |
| [`LocalJsonStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalJsonStore)(path_format[, max_levels]) | Local files store for JSON data: values are read with `json.loads` and written with `json.dumps`. |
| [`LocalPickleStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalPickleStore)(path_format[, max_levels, ...]) | Local files store with pickle serialization: values are any picklable Python object. |
| [`LocalStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalStore) | |
| [`LocalTextStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalTextStore)(path_format[, max_levels]) | Local files store for text data: keys are paths relative to the root, values are `str`. |
| [`PathFormatStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.PathFormatStore)(path_format[, max_levels, mode]) | Local file store using templated relative paths. |
| [`PathFormatStoreWithPrefix`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.PathFormatStoreWithPrefix)(path_format[, ...]) | `PathFormatStore` wrapped in a `Store`, with the root directory available as `_prefix`. |
| [`PickleStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.PickleStore) | |
| [`QuickBinaryStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickBinaryStore)([path_format, max_levels]) | `LocalBinaryStore` with a temporary default root and directories created on write. |
| [`QuickJsonStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickJsonStore)([path_format, max_levels]) | `QuickTextStore` whose values are read with `json.loads` and written with `json.dumps`. |
| [`QuickLocalStoreMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickLocalStoreMixin)([path_format, max_levels]) | A mixin that will choose a path_format if none given, and will automatically create directories on setitem, when missing. |
| [`QuickPickleStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickPickleStore)([path_format, max_levels]) | `LocalPickleStore` with a temporary default root and directories created on write. |
| [`QuickStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickStore) | |
| [`QuickTextStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickTextStore)([path_format, max_levels]) | `LocalTextStore` with a temporary default root and directories created on write. |
| [`RelativeDirPathFormatKeys`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.RelativeDirPathFormatKeys)(path_format[, ...]) | `DirpathFormatKeys` (the folders under a root) wrapped in a `Store` with keys relative to the root. |
| [`RelativePathFormatStore2`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.RelativePathFormatStore2)(path_format[, ...]) | `PathFormatStoreWithPrefix` with keys made relative to the root directory. |
### *class* py2store.stores.local_store.AutoMkDirsOnSetitemMixin
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A mixin that will automatically create directories on setitem, when missing.
### *class* py2store.stores.local_store.AutoMkPathformatMixin(path_format=None, max_levels=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A mixin that will choose a path_format if none given
#### *classmethod* mk_tmp_quick_store_path_format(subpath='')
Path of `subpath` under the class’s folder (`_tmp_dirname`) in the system temp directory.
### *class* py2store.stores.local_store.DirStore(rootdir)
Bases: `Store`
A store for local directories.
Keys are directory names and values are subdirectory DirStores.
```pycon
>>> from py2store import __file__
>>> import os
>>> root = os.path.dirname(__file__)
>>> s = DirStore(root)
>>> assert set(s).issuperset({'stores', 'persisters', 'serializers', 'key_mappers'})
```
### *class* py2store.stores.local_store.LocalBinaryStore(path_format, max_levels=None)
Bases: [`PathFormatPersister`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormatPersister)
Local files store for binary data: like `LocalTextStore`, but values are `bytes`.
```pycon
>>> import tempfile
>>> s = LocalBinaryStore(tempfile.mkdtemp())
>>> s['raw.bin'] = b'ab'
>>> s['raw.bin']
b'ab'
```
### *class* py2store.stores.local_store.LocalJsonStore(path_format, max_levels=None)
Bases: `SimpleJsonMixin`, [`LocalTextStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalTextStore)
Local files store for JSON data: values are read with `json.loads` and written with `json.dumps`.
```pycon
>>> import tempfile
>>> s = LocalJsonStore(tempfile.mkdtemp())
>>> s['conf.json'] = {'a': 1}
>>> s['conf.json']
{'a': 1}
```
### *class* py2store.stores.local_store.LocalPickleStore(path_format, max_levels=None, fix_imports=True, protocol=None, pickle_encoding='ASCII', pickle_errors='strict', \*\*open_kwargs)
Bases: [`PathFormatPersister`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormatPersister)
Local files store with pickle serialization: values are any picklable Python object.
* **Parameters:**
* **path_format** – The root directory, optionally with a `{}` template (see `LocalTextStore`).
* **max_levels** – How many directory levels below the root to include when iterating.
* **fix_imports** – Forwarded to `pickle.dumps` and `pickle.loads`.
* **protocol** – The pickle protocol used when writing.
* **pickle_encoding** – Forwarded to `pickle.loads`.
* **pickle_errors** – Forwarded to `pickle.loads`.
* **\*\*open_kwargs** – Forwarded to `open` when reading and writing files.
* **Raises:**
[**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError) – When unpickling a value needs a module that cannot be imported
(the message names the key).
```pycon
>>> import tempfile
>>> s = LocalPickleStore(tempfile.mkdtemp())
>>> s['obj'] = {'x': [1, 2]}
>>> s['obj']
{'x': [1, 2]}
>>> s.head()
('obj', {'x': [1, 2]})
```
#### *classmethod* for_dill(path_format, max_levels=None, open_kwargs=None, \*args, \*\*kwargs)
Make a store that serializes with `dill` instead of `pickle`; `*args` and `**kwargs` go to `mk_dill_rw_funcs`.
#### head()
Return the first `(key, value)` item, or `None` if the store is empty.
### py2store.stores.local_store.LocalStore
alias of [`QuickPickleStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickPickleStore)
### *class* py2store.stores.local_store.LocalTextStore(path_format, max_levels=None)
Bases: [`PathFormatPersister`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormatPersister)
Local files store for text data: keys are paths relative to the root, values are `str`.
Directories are not created for you: writing under a missing directory raises
`FolderNotFoundError`. Use `QuickTextStore` to have them created on write.
* **Parameters:**
* **path_format** – The root directory, optionally followed by a `{}` template (for example
`'/data/{}.txt'`) that restricts which files under the root are listed
(a key that does not match the template can still be read or written).
* **max_levels** – How many directory levels below the root to include when iterating
(`None` for no limit).
```pycon
>>> import os, tempfile
>>> rootdir = tempfile.mkdtemp()
>>> s = LocalTextStore(rootdir)
>>> len(s)
0
>>> s['hello.txt'] = 'world'
>>> list(s), s['hello.txt'], 'hello.txt' in s
(['hello.txt'], 'world', True)
```
A template filters the listing; it does not change how a key is written:
```pycon
>>> only_txt = LocalTextStore(os.path.join(rootdir, '{}.txt'))
>>> only_txt['notes'] = 'x' # written to rootdir/notes, not rootdir/notes.txt
>>> list(only_txt)
['hello.txt']
```
### *class* py2store.stores.local_store.PathFormatStore(path_format, max_levels=inf, mode='', \*\*open_kwargs)
Bases: [`PathFormatPersister`](_autosummary/py2store.persisters.local_files.html.md#py2store.persisters.local_files.PathFormatPersister), `KvPersister`
Local file store using templated relative paths.
```pycon
>>> from tempfile import gettempdir
>>> import os
>>>
>>> def write_to_key(fullpath_of_relative_path, relative_path, content): # a function to write content in files
... with open(fullpath_of_relative_path(relative_path), 'w') as fp:
... fp.write(content)
>>>
>>> # Preparation: Make a temporary rootdir and write two files in it
>>> rootdir = os.path.join(gettempdir(), 'path_format_store_test' + os.sep)
>>> if not os.path.isdir(rootdir):
... os.mkdir(rootdir)
>>> # recreate directory (remove existing files, delete directory, and re-create it)
>>> for f in os.listdir(rootdir):
... fullpath = os.path.join(rootdir, f)
... if os.path.isfile(fullpath):
... os.remove(os.path.join(rootdir, f))
>>> if os.path.isdir(rootdir):
... os.rmdir(rootdir)
>>> if not os.path.isdir(rootdir):
... os.mkdir(rootdir)
>>>
>>> filepath_of = lambda p: os.path.join(rootdir, p) # a function to get a fullpath from a relative one
>>> # and make two files in this new dir, with some content
>>> write_to_key(filepath_of, 'a', 'foo')
>>> write_to_key(filepath_of, 'b', 'bar')
>>>
>>> # point the obj source to the rootdir
>>> s = PathFormatStore(path_format=rootdir)
>>>
>>> # assert things...
>>> assert s._prefix == rootdir # the _rootdir is the one given in constructor
>>> assert s[filepath_of('a')] == 'foo' # (the filepath for) 'a' contains 'foo'
>>>
>>> # two files under rootdir (as long as the OS didn't create it's own under the hood)
>>> len(s)
2
>>> assert sorted(s) == sorted([filepath_of('a'), filepath_of('b')]) # there's two files in s
>>> filepath_of('a') in s # rootdir/a is in s
True
>>> filepath_of('not_there') in s # rootdir/not_there is not in s
False
>>> filepath_of('not_there') not in s # rootdir/not_there is not in s
True
>>> assert sorted(s.keys()) == sorted([filepath_of('a'), filepath_of('b')]) # the keys (filepaths) of s
>>> sorted(s.values()) # the values of s (contents of files)
['bar', 'foo']
>>> assert sorted(s.items()) == sorted([(filepath_of('a'), 'foo'), (filepath_of('b'), 'bar')]) # the (path, content) items
>>> assert s.get('this key is not there', None) is None # trying to get the val of a non-existing key returns None
>>> s.get('this key is not there', 'some default value') # ... or whatever you say
'some default value'
>>>
>>> # add more files to the same folder
>>> write_to_key(filepath_of, 'this.txt', 'this')
>>> write_to_key(filepath_of, 'that.txt', 'blah')
>>> write_to_key(filepath_of, 'the_other.txt', 'bloo')
>>> # see that you now have 5 files
>>> len(s)
5
>>> # and these files contain values:
>>> sorted(s.values())
['bar', 'blah', 'bloo', 'foo', 'this']
>>>
>>> # but if we make an obj source to only take files whose extension is '.txt'...
>>> s = PathFormatStore(path_format=rootdir + '{}.txt')
>>>
>>> rootdir_2 = os.path.join(gettempdir(), 'obj_source_test_2') # get another rootdir
>>> if not os.path.isdir(rootdir_2):
... os.mkdir(rootdir_2)
>>> filepath_of_2 = lambda p: os.path.join(rootdir_2, p)
>>> # and make two files in this new dir, with some content
>>> write_to_key(filepath_of, 'this.txt', 'this')
>>> write_to_key(filepath_of, 'that.txt', 'blah')
>>> write_to_key(filepath_of, 'the_other.txt', 'bloo')
>>>
>>> ss = PathFormatStore(path_format=rootdir_2 + '{}.txt')
>>>
>>> assert s != ss # though pointing to identical content, o and oo are not equal since the paths are not equal!
```
### *class* py2store.stores.local_store.PathFormatStoreWithPrefix(path_format, max_levels=inf, mode='', \*\*open_kwargs)
Bases: `Store`
`PathFormatStore` wrapped in a `Store`, with the root directory available as `_prefix`.
### py2store.stores.local_store.PickleStore
alias of [`LocalPickleStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalPickleStore)
### *class* py2store.stores.local_store.QuickBinaryStore(path_format=None, max_levels=None)
Bases: [`QuickLocalStoreMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickLocalStoreMixin), [`LocalBinaryStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalBinaryStore)
`LocalBinaryStore` with a temporary default root and directories created on write.
### *class* py2store.stores.local_store.QuickJsonStore(path_format=None, max_levels=None)
Bases: `SimpleJsonMixin`, [`QuickTextStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickTextStore)
`QuickTextStore` whose values are read with `json.loads` and written with `json.dumps`.
### *class* py2store.stores.local_store.QuickLocalStoreMixin(path_format=None, max_levels=None)
Bases: [`AutoMkPathformatMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.AutoMkPathformatMixin), [`AutoMkDirsOnSetitemMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.AutoMkDirsOnSetitemMixin)
A mixin that will choose a path_format if none given,
and will automatically create directories on setitem, when missing.
### *class* py2store.stores.local_store.QuickPickleStore(path_format=None, max_levels=None)
Bases: [`QuickLocalStoreMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickLocalStoreMixin), [`LocalPickleStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalPickleStore)
`LocalPickleStore` with a temporary default root and directories created on write.
This is what `QuickStore` and `LocalStore` name. Without a `path_format` a folder
under the system temp directory is used, and its path is printed.
```pycon
>>> import os, tempfile
>>> s = QuickPickleStore(os.path.join(tempfile.mkdtemp(), 'quick'))
>>> s['deep/er/key'] = [1, 2]
>>> s['deep/er/key'], list(s)
([1, 2], ['deep/er/key'])
```
### py2store.stores.local_store.QuickStore
alias of [`QuickPickleStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickPickleStore)
### *class* py2store.stores.local_store.QuickTextStore(path_format=None, max_levels=None)
Bases: [`QuickLocalStoreMixin`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.QuickLocalStoreMixin), [`LocalTextStore`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.LocalTextStore)
`LocalTextStore` with a temporary default root and directories created on write.
```pycon
>>> import os, tempfile
>>> s = QuickTextStore(os.path.join(tempfile.mkdtemp(), 'sub'))
>>> s['x/y.txt'] = 'z' # sub/ and sub/x/ are created for you
>>> list(s), s['x/y.txt']
(['x/y.txt'], 'z')
```
### *class* py2store.stores.local_store.RelativeDirPathFormatKeys(path_format, max_levels=inf)
Bases: `PrefixRelativizationMixin`, `Store`
`DirpathFormatKeys` (the folders under a root) wrapped in a `Store` with keys relative to the root.
### *class* py2store.stores.local_store.RelativePathFormatStore2(path_format, max_levels=inf, mode='', \*\*open_kwargs)
Bases: `PrefixRelativizationMixin`, [`PathFormatStoreWithPrefix`](_autosummary/py2store.stores.local_store.html.md#py2store.stores.local_store.PathFormatStoreWithPrefix)
`PathFormatStoreWithPrefix` with keys made relative to the root directory.
### py2store.stores.local_store.mk_absolute_path(path_format)
Expand a leading `~` and make a path starting with `.` absolute; other paths are returned unchanged.
### py2store.stores.local_store.mk_tmp_quick_store_dirpath(dirname='')
Path of `dirname` under the system temp directory (`tempfile.gettempdir()`).
# _autosummary/py2store.stores.s3_store.html.md
# py2store.stores.s3_store
Forwards to s3dol.s3_store
# _autosummary/py2store.stores.sql_w_sqlalchemy.html.md
# py2store.stores.sql_w_sqlalchemy
Forwards to sqldol
# _autosummary/py2store.test.html.md
# py2store.test
test files
### Functions
| `djoin`(\*paths) | |
|------------------------|----|
| `minifs_join`(\*paths) | |
### Modules
| [`local_files_test`](_autosummary/py2store.test.local_files_test.html.md#module-py2store.test.local_files_test) | testing local files functionality |
|-----------------------------------------------------------------------------------------------------------|-------------------------------------|
| [`quick_test`](_autosummary/py2store.test.quick_test.html.md#module-py2store.test.quick_test) | a quick test |
| [`util`](_autosummary/py2store.test.util.html.md#module-py2store.test.util) | utils for testing |
# _autosummary/py2store.test.local_files_test.html.md
# py2store.test.local_files_test
testing local files functionality
### Functions
| `test_file_reader`() | |
|------------------------|----|
# _autosummary/py2store.test.quick_test.html.md
# py2store.test.quick_test
a quick test
### Functions
| `test_quick_store`() | |
|------------------------|----|
# _autosummary/py2store.test.util.html.md
# py2store.test.util
utils for testing
### Functions
| [`random_dict_gen`](_autosummary/py2store.test.util.html.md#py2store.test.util.random_dict_gen)([fields, word_size_range, ...]) | Random dict (of strings) generator |
|----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
| [`random_formatted_str_gen`](_autosummary/py2store.test.util.html.md#py2store.test.util.random_formatted_str_gen)([format_string, ...]) | Random formatted string generator |
| [`random_string`](_autosummary/py2store.test.util.html.md#py2store.test.util.random_string)([length, alphabet]) | Same as random_word, but it optimized for strings (5-10% faster for words of length 7, 25-30% faster for words of size 1000) |
| [`random_tuple_gen`](_autosummary/py2store.test.util.html.md#py2store.test.util.random_tuple_gen)([tuple_length, ...]) | Random tuple (of strings) generator |
| [`random_word`](_autosummary/py2store.test.util.html.md#py2store.test.util.random_word)(length, alphabet[, concat_func]) | Make a random word by concatenating randomly drawn elements from alphabet together |
| [`random_word_gen`](_autosummary/py2store.test.util.html.md#py2store.test.util.random_word_gen)([word_size_range, alphabet, n]) | Random string generator |
### py2store.test.util.random_dict_gen(fields=('a', 'b', 'c'), word_size_range=(1, 10), alphabet='abcdefghijklmnopqrstuvwxyz', n=100)
Random dict (of strings) generator
* **Parameters:**
* **fields** – Field names for the random dicts
* **word_size_range** – An int, 2-tuple of ints, or list-like object that defines the choices of word sizes
* **alphabet** – A string or iterable defining the alphabet to draw from
* **n** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – The number of elements the generator will yield
* **Returns:**
Random dict (of strings) generator
### py2store.test.util.random_formatted_str_gen(format_string='root/{}/{}_{}.test', word_size_range=(1, 10), alphabet='abcdefghijklmnopqrstuvwxyz', n=100)
Random formatted string generator
* **Parameters:**
* **format_string** – A format string
* **word_size_range** – An int, 2-tuple of ints, or list-like object that defines the choices of word sizes
* **alphabet** – A string or iterable defining the alphabet to draw from
* **n** – The number of elements the generator will yield
* **Returns:**
Yields random strings of the format defined by format_string
### Examples
# >>> list(random_formatted_str_gen(‘root/{}/{}_{}.test’, (2, 5), ‘abc’, n=5))
[(‘root/acba/bb_abc.test’,),
> (‘root/abcb/cbbc_ca.test’,),
> (‘root/ac/ac_cc.test’,),
> (‘root/aacc/ccbb_ab.test’,),
> (‘root/aab/abb_cbab.test’,)]
```pycon
>>> # The following will be made not random (by restricting the constraints to "no choice"
>>> # ... this is so that we get consistent outputs to assert for the doc test.
>>>
>>> # Example with automatic specification
>>> list(random_formatted_str_gen('root/{}/{}_{}.test', (3, 4), 'a', n=2))
[('root/aaa/aaa_aaa.test',), ('root/aaa/aaa_aaa.test',)]
>>>
>>> # Example with manual specification
>>> list(random_formatted_str_gen('indexed field: {0}: named field: {name}', (2, 3), 'z', n=1))
[('indexed field: zz: named field: zz',)]
```
### py2store.test.util.random_string(length=7, alphabet='abcdefghijklmnopqrstuvwxyz')
Same as random_word, but it optimized for strings
(5-10% faster for words of length 7, 25-30% faster for words of size 1000)
### py2store.test.util.random_tuple_gen(tuple_length=3, word_size_range=(1, 10), alphabet='abcdefghijklmnopqrstuvwxyz', n=100)
Random tuple (of strings) generator
* **Parameters:**
* **tuple_length** – The length of the tuples generated
* **word_size_range** – An int, 2-tuple of ints, or list-like object that defines the choices of word sizes
* **alphabet** – A string or iterable defining the alphabet to draw from
* **n** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – The number of elements the generator will yield
* **Returns:**
Random tuple (of strings) generator
### py2store.test.util.random_word(length, alphabet, concat_func=)
Make a random word by concatenating randomly drawn elements from alphabet together
* **Parameters:**
* **length** – Length of the word
* **alphabet** – Alphabet to draw from
* **concat_func** – The concatenation function (e.g. + for strings and lists)
#### NOTE
Repeated elements in alphabet will have more chances of being drawn.
* **Returns:**
A word (whose type depends on what concatenating elements from alphabet produces).
Not making this a proper doctest because I don’t know how to seed the global random temporarily
```pycon
>>> t = random_word(4, 'abcde'); # e.g. 'acae'
>>> t = random_word(5, ['a', 'b', 'c']); # e.g. 'cabba'
>>> t = random_word(4, [[1, 2, 3], [40, 50], [600], [7000]]); # e.g. [40, 50, 7000, 7000, 1, 2, 3]
>>> t = random_word(4, [1, 2, 3, 4]); # e.g. 13 (because adding numbers...)
>>> # ... sometimes it's what you want:
>>> t = random_word(4, [2 ** x for x in range(8)]); # e.g. 105 (binary combination)
>>> t = random_word(4, [1, 2, 3, 4], concat_func=lambda x, y: str(x) + str(y)); # e.g. '4213'
>>> t = random_word(4, [1, 2, 3, 4], concat_func=lambda x, y: int(str(x) + str(y))); # e.g. 3432
```
### py2store.test.util.random_word_gen(word_size_range=(1, 10), alphabet='abcdefghijklmnopqrstuvwxyz', n=100)
Random string generator
* **Parameters:**
* **word_size_range** – An int, 2-tuple of ints, or list-like object that defines the choices of word sizes
* **alphabet** – A string or iterable defining the alphabet to draw from
* **n** – The number of elements the generator will yield
* **Returns:**
Random string generator
# _autosummary/py2store.trans.html.md
# py2store.trans
Forwards to dol.trans:
Transformation/wrapping tools
# _autosummary/py2store.util.html.md
# py2store.util
Forwards to dol.util:
General util objects
# _autosummary/py2store.utils.affine_conversion.html.md
# py2store.utils.affine_conversion
utils to carry out affine transformations (of indices)
### Functions
| [`get_affine_converter_and_inverse`](_autosummary/py2store.utils.affine_conversion.html.md#py2store.utils.affine_conversion.get_affine_converter_and_inverse)([scale, ...]) | Getting two affine functions with given scale and offset, that are inverse of each other. Namely (for input val)::. |
|---------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
### Classes
| [`AffineConverter`](_autosummary/py2store.utils.affine_conversion.html.md#py2store.utils.affine_conversion.AffineConverter)([scale, offset]) | Getting a callable that will perform an affine conversion. Note, it does it as (val - offset) \* scale. |
|-------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
### *class* py2store.utils.affine_conversion.AffineConverter(scale=1.0, offset=0.0)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Getting a callable that will perform an affine conversion.
Note, it does it as
> (val - offset) \* scale
(Note slope-intercept style (though there is the .from_slope_and_intercept constructor method for that)
Inverse is available through the inv method, performing:
```default
val / scale + offset
```
```pycon
>>> convert = AffineConverter(scale=0.5, offset=1)
>>> convert(0)
-0.5
>>> convert(10)
4.5
>>> convert.inv(4)
9.0
>>> convert.inv(4.5)
10.0
```
### py2store.utils.affine_conversion.get_affine_converter_and_inverse(scale=1, offset=0, source_type_cast=None, target_type_cast=None)
Getting two affine functions with given scale and offset, that are inverse of each other. Namely (for input val):
```default
(val - offset) * scale and val / scale + offset
```
Note this is not “slope intercept” style!!
The source_type_cast and target_type_case (optional), allow the user to specify if these transformations need to
be further cast to a given type.
* **Parameters:**
* **scale**
* **offset**
* **source_type_cast** – function to apply to input
* **target_type_cast** – function to apply to output
* **Returns:**
Two single val functions: affine_converter, inverse_affine_converter
#### NOTE
Code is a lot more complex than the basic operations it performs. The reason was a worry of efficiency since
the functions that are returned are intended to be used in long loops.
#### SEE ALSO
ocore.utils.conversion.AffineConverter
```pycon
>>> affine_converter, inverse_affine_converter = get_affine_converter_and_inverse(scale=0.5,offset=1)
>>> affine_converter(0)
-0.5
>>> affine_converter(10)
4.5
>>> inverse_affine_converter(4)
9.0
>>> inverse_affine_converter(4.5)
10.0
>>> affine_converter, inverse_affine_converter = get_affine_converter_and_inverse(scale=0.5,offset=1,target_type_cast=int)
>>> affine_converter(10)
4
```
# _autosummary/py2store.utils.appendable.html.md
# py2store.utils.appendable
utils to make add append and extend functionality to KV stores
# _autosummary/py2store.utils.attr_dict.html.md
# py2store.utils.attr_dict
a data object layer for object attributes
### Functions
| [`attr_wrap`](_autosummary/py2store.utils.attr_dict.html.md#py2store.utils.attr_dict.attr_wrap)(cls[, name]) | Returns a Mapping class that routes attribute access to keys of mapping. |
|---------------------------------------------------------------------------|----------------------------------------------------------------------------|
| `special_dir`(self) | |
### Classes
| [`AttrMap`](_autosummary/py2store.utils.attr_dict.html.md#py2store.utils.attr_dict.AttrMap)(arg) | A read-only façade for navigating a JSON-like object using attribute notation. |
|-----------------------------------------------------------------|----------------------------------------------------------------------------------|
### *class* py2store.utils.attr_dict.AttrMap(arg)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A read-only façade for navigating a JSON-like object using attribute notation.
Based on Luciano Ramalho’s “Fluent Python” book.
```pycon
>>> t = AttrMap({'a': {'b': 2, 'foo': 'bar'}, 'b': [1,2,3]})
>>> t
AttrMap({'a': {'b': 2, 'foo': 'bar'}, 'b': [1, 2, 3]})
>>> t.a
AttrMap({'b': 2, 'foo': 'bar'})
>>> t.a.foo
'bar'
>>> t.b
[1, 2, 3]
```
#### SEE ALSO
‘AttrDict’ in `dol.sources` module.
### py2store.utils.attr_dict.attr_wrap(cls, name=None)
Returns a Mapping class that routes attribute access to keys of mapping.
```pycon
>>> A = attr_wrap(dict)
>>> t = A({'a_special_attr': 'foo', 'another_attr': 2, # valid identifiers
... 42: [1, 2], '$invalid': 'identifier', 'class': 'is a reserved keyword'}) # not valid identifiers
>>> # verify that we have the attr we want
>>> assert 'a_special_attr' in dir(t)
>>> assert 'another_attr' in dir(t)
>>> # verify that we DO NOT have the attr we DO NOT want
>>> assert 42 not in dir(t)
>>> assert '$invalid' not in dir(t)
>>> assert 'class' not in dir(t)
```
#### SEE ALSO
‘AttrDict’ in `dol.sources` module.
### py2store.utils.attr_dict.iskeyword()
x._\_contains_\_(y) <==> y in x.
# _autosummary/py2store.utils.cache_descriptors.html.md
# py2store.utils.cache_descriptors
descriptors to cache data
### Functions
| [`CachedProperty`](_autosummary/py2store.utils.cache_descriptors.html.md#py2store.utils.cache_descriptors.CachedProperty)(\*args) | CachedProperties. |
|---------------------------------------------------------------------------|---------------------|
### Classes
| [`Lazy`](_autosummary/py2store.utils.cache_descriptors.html.md#py2store.utils.cache_descriptors.Lazy)(func[, name]) | Lazy Attributes. |
|---------------------------------------------------------------------------|---------------------------------------------|
| [`cachedIn`](_autosummary/py2store.utils.cache_descriptors.html.md#py2store.utils.cache_descriptors.cachedIn)(attribute_name) | Cached property with given cache attribute. |
| `readproperty`(func) | |
### py2store.utils.cache_descriptors.CachedProperty(\*args)
CachedProperties.
This is usable directly as a decorator when given names, or when not. Any of these patterns
will work:
* `@CachedProperty`
* `@CachedProperty()`
* `@CachedProperty('n','n2')`
* def thing(self: …; thing = CachedProperty(thing)
* def thing(self: …; thing = CachedProperty(thing, ‘n’)
### *class* py2store.utils.cache_descriptors.Lazy(func, name=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Lazy Attributes.
### *class* py2store.utils.cache_descriptors.cachedIn(attribute_name)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Cached property with given cache attribute.
# _autosummary/py2store.utils.cumul_aggreg_write.html.md
# py2store.utils.cumul_aggreg_write
utils for bulk writing – accumulate, aggregate and write when some condition is met
### Functions
| [`condition_flush_on_every_write`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.condition_flush_on_every_write)(cache) | Boolean function used as flush_cache_condition to anytime the cache is non-empty |
|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| `flush_on_exit`(cls) | |
| `infinite_keycount_kvs`(gen) | |
| `join_byte_values_and_key_as_current_utc_milliseconds`(gen) | |
| `join_string_values_and_key_as_current_utc_milliseconds`(gen) | |
| `key_count`(gen[, start]) | |
| `let_through`(gen) | |
| [`mk_group_aggregator`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.mk_group_aggregator)(item_to_kv[, ...]) | Make a generator transforming function that will (a) make a key for each given item, (b) group all items according to the key |
| [`mk_group_aggregator_with_key_func`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.mk_group_aggregator_with_key_func)(item_to_key) | Make a generator transforming function that will (a) make a key for each given item, (b) group all items according to the key |
| `mk_kv_from_keygen`([keygen]) | |
### Classes
| [`CumulAggregWrite`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.CumulAggregWrite)(store[, cache_to_kv, mk_cache]) | |
|-----------------------------------------------------------------------------------------------------|----|
| [`CumulAggregWriteKvItems`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.CumulAggregWriteKvItems)(store) | |
| [`CumulAggregWriteWithAutoFlush`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.CumulAggregWriteWithAutoFlush)(store[, ...]) | |
### *class* py2store.utils.cumul_aggreg_write.CumulAggregWrite(store, cache_to_kv=.aggregate>, mk_cache=)
Bases: [`CumulAggregWrite`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.CumulAggregWrite)
### *class* py2store.utils.cumul_aggreg_write.CumulAggregWriteKvItems(store)
Bases: [`CumulAggregWrite`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.CumulAggregWrite)
### *class* py2store.utils.cumul_aggreg_write.CumulAggregWriteWithAutoFlush(store, cache_to_kv=.aggregate>, mk_cache=, flush_cache_condition=)
Bases: [`CumulAggregWrite`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#py2store.utils.cumul_aggreg_write.CumulAggregWrite)
### py2store.utils.cumul_aggreg_write.condition_flush_on_every_write(cache)
Boolean function used as flush_cache_condition to anytime the cache is non-empty
### py2store.utils.cumul_aggreg_write.mk_group_aggregator(item_to_kv, aggregator_op=, initial=)
Make a generator transforming function that will
(a) make a key for each given item,
(b) group all items according to the key
* **Parameters:**
* **item_to_kv** – Function taking an item and returning the `(key, value)` pair to group by key.
* **aggregator_op** – The aggregation binary function that is used to aggregate two items together.
The function is used as is by the functools.reduce, applied to the sequence of items that were collected for
a given group
* **initial** – The “empty” element to start the reduce (aggregation) with, if necessary.
* **Returns:**
A function taking an iterable of items and yielding `(key, aggregate)` pairs, one per key.
```pycon
>>> # Collect words (as a csv string), grouped by the lower case of the first letter
>>> ag = mk_group_aggregator(lambda item: (item[0].lower(), item),
... aggregator_op=lambda x, y: ', '.join([x, y]))
>>> list(ag(['apple', 'bananna', 'Airplane']))
[('a', 'apple, Airplane'), ('b', 'bananna')]
>>> # Collect (and concatinate) characters according to their ascii value modulo 3
>>> ag = mk_group_aggregator(lambda item: (item['age'], item['thing']),
... aggregator_op=lambda x, y: x + [y],
... initial=[])
>>> list(ag([{'age': 0, 'thing': 'new'}, {'age': 42, 'thing': 'every'}, {'age': 0, 'thing': 'just born'}]))
[(0, ['new', 'just born']), (42, ['every'])]
```
### py2store.utils.cumul_aggreg_write.mk_group_aggregator_with_key_func(item_to_key, aggregator_op=, initial=)
Make a generator transforming function that will
(a) make a key for each given item,
(b) group all items according to the key
* **Parameters:**
* **item_to_key** – Function that takes an item of the generator and outputs the key that should be used to group items
* **aggregator_op** – The aggregation binary function that is used to aggregate two items together.
The function is used as is by the functools.reduce, applied to the sequence of items that were collected for
a given group
* **initial** – The “empty” element to start the reduce (aggregation) with, if necessary.
* **Returns:**
A function taking an iterable of items and yielding `(key, aggregate)` pairs, one per key.
```pycon
>>> # Collect words (as a csv string), grouped by the lower case of the first letter
>>> ag = mk_group_aggregator_with_key_func(lambda item: item[0].lower(),
... aggregator_op=lambda x, y: ', '.join([x, y]))
>>> list(ag(['apple', 'bananna', 'Airplane']))
[('a', 'apple, Airplane'), ('b', 'bananna')]
>>>
>>> # Collect (and concatenate) characters according to their ascii value modulo 3
... ag = mk_group_aggregator_with_key_func(lambda item: (ord(item) % 3))
>>> list(ag('abcdefghijklmnop'))
[(1, 'adgjmp'), (2, 'behkn'), (0, 'cfilo')]
>>>
>>> # sum all even and odd number separately
... ag = mk_group_aggregator_with_key_func(lambda item: (item % 2))
>>> list(ag([1, 2, 3, 4, 5])) # sum of evens is 6, and sum of odds is 9
[(1, 9), (0, 6)]
>>>
>>> # if we wanted to collect all odds and evens, we'd need a different aggregator and initial
... ag = mk_group_aggregator_with_key_func(lambda item: (item % 2), aggregator_op=lambda x, y: x + [y], initial=[])
>>> list(ag([1, 2, 3, 4, 5]))
[(1, [1, 3, 5]), (0, [2, 4])]
```
# _autosummary/py2store.utils.explicit.html.md
# py2store.utils.explicit
utils to make stores based on a the input data itself
# _autosummary/py2store.utils.glom.html.md
# py2store.utils.glom
*glom is a util to extract stuff from nested structures.*
It’s one of those excellent utils that I’ve written many times, but never got quite right.
Mahmoud Hashemi got it right.
* **BEGIN LICENSE:**
Copyright (c) 2018, Mahmoud Hashemi
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
> * Redistributions of source code must retain the above copyright
> notice, this list of conditions and the following disclaimer.
> * Redistributions in binary form must reproduce the above
> copyright notice, this list of conditions and the following
> disclaimer in the documentation and/or other materials provided
> with the distribution.
> * The names of the contributors may not be used to endorse or
> promote products derived from this software without specific
> prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
“AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
* **END LICENSE:**
Now, at the time of writing this, I’ve already transformed it to bend it to my liking.
At some point it may become something else, but I wanted there to be a trace of what my seed was.
Though I can’t promise I’ll maintain the same functionality as I transform this module, here’s
a tutorial on how to use it in it’s original form:
```default
https://glom.readthedocs.io/en/latest/
```
I only took the main (core) module from the glom project.
Here’s the original docs of this glom module.
If there was ever a Python example of “big things come in small
packages”, `glom` might be it.
The `glom` package has one central entrypoint,
`glom.glom()`. Everything else in the package revolves around that
one function.
A couple of conventional terms you’ll see repeated many times below:
* **target** - glom is built to work on any data, so we simply
refer to the object being accessed as the *“target”*
* **spec** - *(aka “glomspec”, short for specification)* The
accompanying template used to specify the structure of the return
value.
Now that you know the terms, let’s take a look around glom’s powerful
semantics.
### Functions
| [`glom`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom)(target, spec, \*\*kwargs) | Access or construct a value from a given *target* based on the specification declared by *spec*. |
|------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`is_iterable`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.is_iterable)(x) | Similar in nature to [`callable()`](https://docs.python.org/3/builtins/functions.html#callable), `is_iterable` returns `True` if an object is iterable, `False` if not. |
| [`make_sentinel`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.make_sentinel)([name, var_name]) | Creates and returns a new **instance** of a new class, suitable for usage as a "sentinel", a kind of singleton often used to indicate a value is missing when `None` is a valid input. |
| [`register`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.register)(target_type, \*\*kwargs) | Register *target_type* so `glom()` will know how to handle instances of that type as targets. |
| [`register_op`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.register_op)(op_name, \*\*kwargs) | For extension authors needing to add operations beyond the builtin 'get' and 'iterate' to the default scope. |
### Classes
| [`Auto`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Auto)([spec]) | Switch to Auto mode (the default) |
|-------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`Call`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Call)([func, args, kwargs]) | [`Call`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Call) specifies when a target should be passed to a function, *func*. |
| [`Check`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Check)([spec]) | Check objects are used to make assertions about the target data, and either pass through the data or raise exceptions if there is a problem. |
| [`Coalesce`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Coalesce)(\*subspecs, \*\*kwargs) | Coalesce objects specify fallback behavior for a list of subspecs. |
| [`Fill`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Fill)([spec]) | A specifier type which switches to glom into "fill-mode". |
| [`Glommer`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Glommer)(\*\*kwargs) | All the wholesome goodness that it takes to make glom work. |
| [`Inspect`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Inspect)(\*a, \*\*kw) | The `Inspect` specifier type provides a way to get visibility into glom's evaluation of a specification, enabling debugging of those tricky problems that may arise with unexpected data. |
| [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke)(func) | Specifier type designed for easy invocation of callables from glom. |
| [`Let`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Let)(\*\*kw) | This specifier type assigns variables to the scope. |
| [`Literal`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Literal)(value) | Literal objects specify literal values in rare cases when part of the spec should not be interpreted as a glommable subspec. |
| [`Path`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Path)(\*path_parts) | Path objects specify explicit paths when the default `'a.b.c'`-style general access syntax won't work or isn't desirable. |
| [`Spec`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Spec)(spec[, scope]) | Spec objects serve three purposes, here they are, roughly ordered by utility: |
| [`TType`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.TType)() | `T`, short for "target". |
| [`TargetRegistry`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.TargetRegistry)([register_default_types]) | responsible for registration of target types for iteration and attribute walking |
### Exceptions
| [`CheckError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.CheckError)(msgs, check, path) | This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype is raised when target data fails to pass a [`Check`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Check)'s specified validation. |
|-------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`CoalesceError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.CoalesceError)(coal_obj, skipped, path) | This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype is raised from within a [`Coalesce`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Coalesce) spec's processing, when none of the subspecs match and no default is provided. |
| [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) | The base exception for all the errors that might be raised from [`glom()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom) processing logic. |
| [`PathAccessError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.PathAccessError)(exc, path, part_idx) | This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype represents a failure to access an attribute as dictated by the spec. |
| [`UnregisteredTarget`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.UnregisteredTarget)(op, target_type, ...) | This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype is raised when a spec calls for an unsupported action on a target type. |
### *class* py2store.utils.glom.Auto(spec=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Switch to Auto mode (the default)
### *class* py2store.utils.glom.Call(func=None, args=None, kwargs=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
[`Call`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Call) specifies when a target should be passed to a function,
*func*.
[`Call`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Call) is similar to [`partial()`](https://docs.python.org/3/library/functools.html#functools.partial) in that
it is no more powerful than `lambda` or other functions, but
it is designed to be more readable, with a better `repr`.
* **Parameters:**
**func** (*callable*) – a function or other callable to be called with
the target
[`Call`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Call) combines well with `T` to construct objects. For
instance, to generate a dict and then pass it to a constructor:
```pycon
>>> class ExampleClass(object):
... def __init__(self, attr):
... self.attr = attr
...
>>> target = {'attr': 3.14}
>>> glom(target, Call(ExampleClass, kwargs=T)).attr
3.14
```
This does the same as `glom(target, lambda target:
ExampleClass(\*\*target))`, but it’s easy to see which one reads
better.
#### NOTE
`Call` is mostly for functions. Use a `T` object
if you need to call a method.
#### WARNING
[`Call`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Call) has a successor with a fuller-featured API, new
in 19.3.0: the [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke) specifier type.
#### glomit(target, scope)
run against the current target
### *class* py2store.utils.glom.Check(spec=T, \*\*kwargs)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Check objects are used to make assertions about the target data,
and either pass through the data or raise exceptions if there is a
problem.
If any check condition fails, a `CheckError` is raised.
* **Parameters:**
* **spec** – a sub-spec to extract the data to which other assertions will
be checked (defaults to applying checks to the target itself)
* **type** – a type or sequence of types to be checked for exact match
* **equal_to** – a value to be checked for equality match (“==”)
* **validate** – a callable or list of callables, each representing a
check condition. If one or more return False or raise an
exception, the Check will fail.
* **instance_of** – a type or sequence of types to be checked with isinstance()
* **one_of** – an iterable of values, any of which can match the target (“in”)
* **default** – an optional default value to replace the value when the check fails
(if default is not specified, GlomCheckError will be raised)
Aside from *spec*, all arguments are keyword arguments. Each
argument, except for *default*, represent a check
condition. Multiple checks can be passed, and if all check
conditions are left unset, Check defaults to performing a basic
truthy check on the value.
### *exception* py2store.utils.glom.CheckError(msgs, check, path)
Bases: [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError)
This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype is raised when target data fails to
pass a [`Check`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Check)’s specified validation.
An uncaught `CheckError` looks like this:
```default
>>> target = {'a': {'b': 'c'}}
>>> glom(target, {'b': ('a.b', Check(type=int))})
Traceback (most recent call last):
...
glom.CheckError: target at path ['a.b'] failed check, got error: "expected type to be 'int', found type 'str'"
```
If the `Check` contains more than one condition, there may be
more than one error message. The string rendition of the
`CheckError` will include all messages.
You can also catch the `CheckError` and programmatically access
messages through the `msgs` attribute on the `CheckError`
instance.
#### NOTE
As of 2018-07-05 (glom v18.2.0), the validation subsystem is
still very new. Exact error message formatting may be enhanced
in future releases.
### *class* py2store.utils.glom.Coalesce(\*subspecs, \*\*kwargs)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Coalesce objects specify fallback behavior for a list of
subspecs.
Subspecs are passed as positional arguments, and keyword arguments
control defaults. Each subspec is evaluated in turn, and if none
match, a [`CoalesceError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.CoalesceError) is raised, or a default is returned,
depending on the options used.
#### NOTE
This operation may seem very familar if you have experience with
[SQL](https://en.wikipedia.org/w/index.php?title=Null_(SQL)&oldid=833093792#COALESCE) or even [C# and others](https://en.wikipedia.org/w/index.php?title=Null_coalescing_operator&oldid=839493322#C#).
In practice, this fallback behavior’s simplicity is only surpassed
by its utility:
```pycon
>>> target = {'c': 'd'}
>>> glom(target, Coalesce('a', 'b', 'c'))
'd'
```
glom tries to get `'a'` from `target`, but gets a
KeyError. Rather than raise a `PathAccessError` as usual,
glom *coalesces* into the next subspec, `'b'`. The process
repeats until it gets to `'c'`, which returns our value,
`'d'`. If our value weren’t present, we’d see:
```pycon
>>> target = {}
>>> glom(target, Coalesce('a', 'b'))
Traceback (most recent call last):
...
glom.CoalesceError: no valid values found. Tried ('a', 'b') and got (PathAccessError, PathAccessError) (at path [])
```
Same process, but because `target` is empty, we get a
[`CoalesceError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.CoalesceError). If we want to avoid an exception, and we
know which value we want by default, we can set *default*:
```pycon
>>> target = {}
>>> glom(target, Coalesce('a', 'b', 'c'), default='d-fault')
'd-fault'
```
`'a'`, `'b'`, and `'c'` weren’t present so we got `'d-fault'`.
* **Parameters:**
* **subspecs** – One or more glommable subspecs
* **default** – A value to return if no subspec results in a valid value
* **default_factory** – A callable whose result will be returned as a default
* **skip** – A value, tuple of values, or predicate function
representing values to ignore
* **skip_exc** – An exception or tuple of exception types to catch and
move on to the next subspec. Defaults to [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError), the
parent type of all glom runtime exceptions.
If all subspecs produce skipped values or exceptions, a
[`CoalesceError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.CoalesceError) will be raised. For more examples, check out
the tutorial, which makes extensive use of Coalesce.
### *exception* py2store.utils.glom.CoalesceError(coal_obj, skipped, path)
Bases: [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError)
This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype is raised from within a
[`Coalesce`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Coalesce) spec’s processing, when none of the subspecs
match and no default is provided.
The exception object itself keeps track of several values which
may be useful for processing:
* **Parameters:**
* **coal_obj** ([*Coalesce*](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Coalesce)) – The original failing spec, see
[`Coalesce`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Coalesce)’s docs for details.
* **skipped** ([*list*](https://docs.python.org/3/builtins/stdtypes.html#list)) – A list of ignored values and exceptions, in the
order that their respective subspecs appear in the original
*coal_obj*.
* **path** – Like many GlomErrors, this exception knows the path at
which it occurred.
```pycon
>>> target = {}
>>> glom(target, Coalesce('a', 'b'))
Traceback (most recent call last):
...
glom.CoalesceError: no valid values found. Tried ('a', 'b') and got (PathAccessError, PathAccessError) ...
```
### *class* py2store.utils.glom.Fill(spec=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A specifier type which switches to glom into “fill-mode”. For the
spec contained within the Fill, glom will only interpret explicit
specifier types (including T objects). Whereas the default mode
has special interpretations for each of these builtins, fill-mode
takes a lighter touch, making Fill great for “filling out” Python
literals, like tuples, dicts, sets, and lists.
```pycon
>>> target = {'data': [0, 2, 4]}
>>> spec = Fill((T['data'][2], T['data'][0]))
>>> glom(target, spec)
(4, 0)
```
As you can see, glom’s usual built-in tuple item chaining behavior
has switched into a simple tuple constructor.
(Sidenote for Lisp fans: Fill is like glom’s quasi-quoting.)
### *exception* py2store.utils.glom.GlomError
Bases: [`Exception`](https://docs.python.org/3/builtins/exceptions.html#Exception)
The base exception for all the errors that might be raised from
[`glom()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom) processing logic.
By default, exceptions raised from within functions passed to glom
(e.g., `len`, `sum`, any `lambda`) will not be wrapped in a
GlomError.
### *class* py2store.utils.glom.Glommer(\*\*kwargs)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
All the wholesome goodness that it takes to make glom work. This
type mostly serves to encapsulate the type registration context so
that advanced uses of glom don’t need to worry about stepping on
each other’s toes.
Glommer objects are lightweight and, once instantiated, provide
the [`glom()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom) method we know and love:
```pycon
>>> glommer = Glommer()
>>> glommer.glom({}, 'a.b.c', default='d')
'd'
>>> Glommer().glom({'vals': list(range(3))}, ('vals', len))
3
```
Instances also provide [`register()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Glommer.register) method for
localized control over type handling.
* **Parameters:**
**register_default_types** ([*bool*](https://docs.python.org/3/builtins/functions.html#bool)) – Whether or not to enable the
handling behaviors of the default [`glom()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom). These
default actions include dict access, list and iterable
iteration, and generic object attribute access. Defaults to
True.
#### register(target_type, \*\*kwargs)
Register *target_type* so `glom()` will
know how to handle instances of that type as targets.
* **Parameters:**
* **target_type** ([*type*](https://docs.python.org/3/builtins/functions.html#type)) – A type expected to appear in a glom()
call target
* **get** (*callable*) – A function which takes a target object and
a name, acting as a default accessor. Defaults to
[`getattr()`](https://docs.python.org/3/builtins/functions.html#getattr).
* **iterate** (*callable*) – A function which takes a target object
and returns an iterator. Defaults to [`iter()`](https://docs.python.org/3/builtins/functions.html#iter) if
*target_type* appears to be iterable.
* **exact** ([*bool*](https://docs.python.org/3/builtins/functions.html#bool)) – Whether or not to match instances of subtypes
of *target_type*.
#### NOTE
The module-level [`register()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.register) function affects the
module-level [`glom()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom) function’s behavior. If this
global effect is undesirable for your application, or
you’re implementing a library, consider instantiating a
[`Glommer`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Glommer) instance, and using the
[`register()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Glommer.register) and `Glommer.glom()`
methods instead.
### *class* py2store.utils.glom.Inspect(\*a, \*\*kw)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The `Inspect` specifier type provides a way to get
visibility into glom’s evaluation of a specification, enabling
debugging of those tricky problems that may arise with unexpected
data.
`Inspect` can be inserted into an existing spec in one of two
ways. First, as a wrapper around the spec in question, or second,
as an argument-less placeholder wherever a spec could be.
`Inspect` supports several modes, controlled by
keyword arguments. Its default, no-argument mode, simply echos the
state of the glom at the point where it appears:
```pycon
>>> target = {'a': {'b': {}}}
>>> val = glom(target, Inspect('a.b')) # wrapping a spec
---
path: ['a.b']
target: {'a': {'b': {}}}
output: {}
---
```
Debugging behavior aside, `Inspect` has no effect on
values in the target, spec, or result.
* **Parameters:**
* **echo** ([*bool*](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to print the path, target, and output of
each inspected glom. Defaults to True.
* **recursive** ([*bool*](https://docs.python.org/3/builtins/functions.html#bool)) – Whether or not the Inspect should be applied
at every level, at or below the spec that it wraps. Defaults
to False.
* **breakpoint** ([*bool*](https://docs.python.org/3/builtins/functions.html#bool)) – This flag controls whether a debugging prompt
should appear before evaluating each inspected spec. Can also
take a callable. Defaults to False.
* **post_mortem** ([*bool*](https://docs.python.org/3/builtins/functions.html#bool)) – This flag controls whether exceptions
should be caught and interactively debugged with [`pdb`](https://docs.python.org/3/library/pdb.html#module-pdb) on
inspected specs.
All arguments above are keyword-only to avoid overlap with a
wrapped spec.
#### NOTE
Just like `pdb.set_trace()`, be careful about leaving stray
`Inspect()` instances in production glom specs.
### *class* py2store.utils.glom.Invoke(func)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Specifier type designed for easy invocation of callables from glom.
* **Parameters:**
**func** (*callable*) – A function or other callable object.
`Invoke` is similar to [`functools.partial()`](https://docs.python.org/3/library/functools.html#functools.partial), but with the
ability to set up a “templated” call which interleaves constants and
glom specs.
For example, the following creates a spec which can be used to
check if targets are integers:
```pycon
>>> is_int = Invoke(isinstance).specs(T).constants(int)
>>> glom(5, is_int)
True
```
And this composes like any other glom spec:
```pycon
>>> target = [7, object(), 9]
>>> glom(target, [is_int])
[True, False, True]
```
Another example, mixing positional and keyword arguments:
```pycon
>>> spec = Invoke(sorted).specs(T).constants(key=int, reverse=True)
>>> target = ['10', '5', '20', '1']
>>> glom(target, spec)
['20', '10', '5', '1']
```
Invoke also helps with evaluating zero-argument functions:
```pycon
>>> glom(target={}, spec=Invoke(int))
0
```
(A trivial example, but from timestamps to UUIDs, zero-arg calls do come up!)
#### NOTE
`Invoke` is mostly for functions, object construction, and callable
objects. For calling methods, consider the `T` object.
#### constants(\*a, \*\*kw)
Returns a new [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke) spec, with the provided positional
and keyword argument values stored for passing to the
underlying function.
```pycon
>>> spec = Invoke(T).constants(5)
>>> glom(range, (spec, list))
[0, 1, 2, 3, 4]
```
Subsequent positional arguments are appended:
```pycon
>>> spec = Invoke(T).constants(2).constants(10, 2)
>>> glom(range, (spec, list))
[2, 4, 6, 8]
```
Keyword arguments also work as one might expect:
```pycon
>>> round_2 = Invoke(round).constants(ndigits=2).specs(T)
>>> glom(3.14159, round_2)
3.14
```
[`constants()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke.constants) and other [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke)
methods may be called multiple times, just remember that every
call returns a new spec.
#### *classmethod* specfunc(spec)
Creates an [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke) instance where the function is
indicated by a spec.
```pycon
>>> spec = Invoke.specfunc('func').constants(5)
>>> glom({'func': range}, (spec, list))
[0, 1, 2, 3, 4]
```
#### specs(\*a, \*\*kw)
Returns a new [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke) spec, with the provided positional
and keyword arguments stored to be interpreted as specs, with
the results passed to the underlying function.
```pycon
>>> spec = Invoke(range).specs('value')
>>> glom({'value': 5}, (spec, list))
[0, 1, 2, 3, 4]
```
Subsequent positional arguments are appended:
```pycon
>>> spec = Invoke(range).specs('start').specs('end', 'step')
>>> target = {'start': 2, 'end': 10, 'step': 2}
>>> glom(target, (spec, list))
[2, 4, 6, 8]
```
Keyword arguments also work as one might expect:
```pycon
>>> multiply = lambda x, y: x * y
>>> times_3 = Invoke(multiply).constants(y=3).specs(x='value')
>>> glom({'value': 5}, times_3)
15
```
[`specs()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke.specs) and other [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke)
methods may be called multiple times, just remember that every
call returns a new spec.
#### star(args=None, kwargs=None)
Returns a new [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke) spec, with *args* and/or *kwargs*
specs set to be “starred” or “star-starred” (respectively)
```pycon
>>> import os.path
>>> spec = Invoke(os.path.join).star(args='path')
>>> target = {'path': ['path', 'to', 'dir']}
>>> glom(target, spec)
'path/to/dir'
```
* **Parameters:**
* **args** (*spec*) – A spec to be evaluated and “starred” into the
underlying function.
* **kwargs** (*spec*) – A spec to be evaluated and “star-starred” into
the underlying function.
One or both of the above arguments should be set.
The [`star()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke.star), like other [`Invoke`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Invoke)
methods, may be called multiple times. The *args* and *kwargs*
will be stacked in the order in which they are provided.
### *class* py2store.utils.glom.Let(\*\*kw)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
This specifier type assigns variables to the scope.
```pycon
>>> target = {'data': {'val': 9}}
>>> spec = (Let(value=T['data']['val']), {'val': S['value']})
>>> glom(target, spec)
{'val': 9}
```
### *class* py2store.utils.glom.Literal(value)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Literal objects specify literal values in rare cases when part of
the spec should not be interpreted as a glommable
subspec. Wherever a Literal object is encountered in a spec, it is
replaced with its wrapped *value* in the output.
```pycon
>>> target = {'a': {'b': 'c'}}
>>> spec = {'a': 'a.b', 'readability': Literal('counts')}
>>> pprint(glom(target, spec))
{'a': 'c', 'readability': 'counts'}
```
Instead of accessing `'counts'` as a key like it did with
`'a.b'`, `glom()` just unwrapped the literal and
included the value.
`Literal` takes one argument, the literal value that should appear
in the glom output.
This could also be achieved with a callable, e.g., `lambda x:
'literal_string'` in the spec, but using a `Literal`
object adds explicitness, code clarity, and a clean [`repr()`](https://docs.python.org/3/builtins/functions.html#repr).
### *class* py2store.utils.glom.Path(\*path_parts)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Path objects specify explicit paths when the default
`'a.b.c'`-style general access syntax won’t work or isn’t
desirable. Use this to wrap ints, datetimes, and other valid
keys, as well as strings with dots that shouldn’t be expanded.
```pycon
>>> target = {'a': {'b': 'c', 'd.e': 'f', 2: 3}}
>>> glom(target, Path('a', 2))
3
>>> glom(target, Path('a', 'd.e'))
'f'
```
Paths can be used to join together other Path objects, as
well as `T` objects:
```pycon
>>> Path(T['a'], T['b'])
T['a']['b']
>>> Path(Path('a', 'b'), Path('c', 'd'))
Path('a', 'b', 'c', 'd')
```
Paths also support indexing and slicing, with each access
returning a new Path object:
```pycon
>>> path = Path('a', 'b', 1, 2)
>>> path[0]
Path('a')
>>> path[-2:]
Path(1, 2)
```
#### from_t()
return the same path but starting from T
#### *classmethod* from_text(text)
Make a Path from .-delimited text:
```pycon
>>> Path.from_text('a.b.c')
Path('a', 'b', 'c')
```
#### items()
Returns a tuple of (operation, value) pairs.
```pycon
>>> Path(T.a.b, 'c', T['d']).items()
(('.', 'a'), ('.', 'b'), ('P', 'c'), ('[', 'd'))
```
#### values()
Returns a tuple of values referenced in this path.
```pycon
>>> Path(T.a.b, 'c', T['d']).values()
('a', 'b', 'c', 'd')
```
### *exception* py2store.utils.glom.PathAccessError(exc, path, part_idx)
Bases: [`AttributeError`](https://docs.python.org/3/builtins/exceptions.html#AttributeError), [`KeyError`](https://docs.python.org/3/builtins/exceptions.html#KeyError), [`IndexError`](https://docs.python.org/3/builtins/exceptions.html#IndexError), [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError)
This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype represents a failure to access an
attribute as dictated by the spec. The most commonly-seen error
when using glom, it maintains a copy of the original exception and
produces a readable error message for easy debugging.
If you see this error, you may want to:
> * Check the target data is accurate using `Inspect`
> * Catch the exception and return a semantically meaningful error message
> * Use `glom.Coalesce` to specify a default
> * Use the top-level `default` kwarg on `glom()`
In any case, be glad you got this error and not the one it was
wrapping!
* **Parameters:**
* **exc** ([*Exception*](https://docs.python.org/3/builtins/exceptions.html#Exception)) – The error that arose when we tried to access
*path*. Typically an instance of KeyError, AttributeError,
IndexError, or TypeError, and sometimes others.
* **path** ([*Path*](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Path)) – The full Path glom was in the middle of accessing
when the error occurred.
* **part_idx** ([*int*](https://docs.python.org/3/builtins/functions.html#int)) – The index of the part of the *path* that caused
the error.
```pycon
>>> target = {'a': {'b': None}}
>>> glom(target, 'a.b.c')
Traceback (most recent call last):
...
glom.PathAccessError: could not access 'c', part 2 of Path('a', 'b', 'c'), got error: ...
```
### *class* py2store.utils.glom.Spec(spec, scope=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Spec objects serve three purposes, here they are, roughly ordered
by utility:
> 1. As a form of compiled or “curried” glom call, similar to
> Python’s built-in [`re.compile()`](https://docs.python.org/3/library/re.html#re.compile).
> 2. A marker as an object as representing a spec rather than a
> literal value in certain cases where that might be ambiguous.
> 3. A way to update the scope within another Spec.
In the second usage, Spec objects are the complement to
`Literal`, wrapping a value and marking that it
should be interpreted as a glom spec, rather than a literal value.
This is useful in places where it would be interpreted as a value
by default. (Such as T[key], Call(func) where key and func are
assumed to be literal values and not specs.)
* **Parameters:**
* **spec** – The glom spec.
* **scope** ([*dict*](https://docs.python.org/3/builtins/stdtypes.html#dict)) – additional values to add to the scope when
evaluating this Spec
### *class* py2store.utils.glom.TType
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
`T`, short for “target”. A singleton object that enables
object-oriented expression of a glom specification.
#### NOTE
`T` is a singleton, and does not need to be constructed.
Basically, think of `T` as your data’s stunt double. Everything
that you do to `T` will be recorded and executed during the
[`glom()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom) call. Take this example:
```pycon
>>> spec = T['a']['b']['c']
>>> target = {'a': {'b': {'c': 'd'}}}
>>> glom(target, spec)
'd'
```
So far, we’ve relied on the `'a.b.c'`-style shorthand for
access, or used the `Path` objects, but if you want
to explicitly do attribute and key lookups, look no further than
`T`.
But T doesn’t stop with unambiguous access. You can also call
methods and perform almost any action you would with a normal
object:
```pycon
>>> spec = ('a', (T['b'].items(), list)) # reviewed below
>>> glom(target, spec)
[('c', 'd')]
```
A `T` object can go anywhere in the spec. As seen in the example
above, we access `'a'`, use a `T` to get `'b'` and iterate
over its `items`, turning them into a `list`.
You can even use `T` with `Call` to construct objects:
```pycon
>>> class ExampleClass(object):
... def __init__(self, attr):
... self.attr = attr
...
>>> target = {'attr': 3.14}
>>> glom(target, Call(ExampleClass, kwargs=T)).attr
3.14
```
On a further note, while `lambda` works great in glom specs, and
can be very handy at times, `T` and `Call`
eliminate the need for the vast majority of `lambda` usage with
glom.
Unlike `lambda` and other functions, `T` roundtrips
beautifully and transparently:
```pycon
>>> T['a'].b['c']('success')
T['a'].b['c']('success')
```
`T`-related access errors raise a `PathAccessError`
during the `glom()` call.
#### NOTE
While `T` is clearly useful, powerful, and here to stay, its
semantics are still being refined. Currently, operations beyond
method calls and attribute/item access are considered
experimental and should not be relied upon.
### *class* py2store.utils.glom.TargetRegistry(register_default_types=True)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
responsible for registration of target types for iteration
and attribute walking
#### get_handler(op, obj, path=None, raise_exc=True)
for an operation and object **instance**, obj, return the
closest-matching handler function, raising UnregisteredTarget
if no handler can be found for *obj* (or False if
raise_exc=False)
#### register_op(op_name, auto_func=None, exact=False)
add operations beyond the builtins (‘get’ and ‘iterate’ at the time
of writing).
auto_func is a function that when passed a type, returns a
handler associated with op_name if it’s supported, or False if
it’s not.
See glom.core.register_op() for the global version used by
extensions.
### *exception* py2store.utils.glom.UnregisteredTarget(op, target_type, type_map, path)
Bases: [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError)
This [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError) subtype is raised when a spec calls for an
unsupported action on a target type. For instance, trying to
iterate on an non-iterable target:
```pycon
>>> glom(object(), ['a.b.c'])
Traceback (most recent call last):
...
glom.UnregisteredTarget: target type 'object' not registered for 'iterate', expected one of registered types: (...)
```
It should be noted that this is a pretty uncommon occurrence in
production glom usage. See the setup-and-registration
section for details on how to avoid this error.
An UnregisteredTarget takes and tracks a few values:
* **Parameters:**
* **op** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the operation being performed (‘get’ or ‘iterate’)
* **target_type** ([*type*](https://docs.python.org/3/builtins/functions.html#type)) – The type of the target being processed.
* **type_map** ([*dict*](https://docs.python.org/3/builtins/stdtypes.html#dict)) – A mapping of target types that do support this operation
* **path** – The path at which the error occurred.
### py2store.utils.glom.glom(target, spec, \*\*kwargs)
Access or construct a value from a given *target* based on the
specification declared by *spec*.
Accessing nested data, aka deep-get:
```pycon
>>> target = {'a': {'b': 'c'}}
>>> glom(target, 'a.b')
'c'
```
Here the *spec* was just a string denoting a path,
`'a.b.`. As simple as it should be. The next example shows
how to use nested data to access many fields at once, and make
a new nested structure.
Constructing, or restructuring more-complicated nested data:
```pycon
>>> target = {'a': {'b': 'c', 'd': 'e'}, 'f': 'g', 'h': [0, 1, 2]}
>>> spec = {'a': 'a.b', 'd': 'a.d', 'h': ('h', [lambda x: x * 2])}
>>> output = glom(target, spec)
>>> pprint(output)
{'a': 'c', 'd': 'e', 'h': [0, 2, 4]}
```
`glom` also takes a keyword-argument, *default*. When set,
if a `glom` operation fails with a [`GlomError`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.GlomError), the
*default* will be returned, very much like
[`dict.get()`](https://docs.python.org/3/builtins/stdtypes.html#dict.get):
```pycon
>>> glom(target, 'a.xx', default='nada')
'nada'
```
The *skip_exc* keyword argument controls which errors should
be ignored.
```pycon
>>> glom({}, lambda x: 100.0 / len(x), default=0.0, skip_exc=ZeroDivisionError)
0.0
```
* **Parameters:**
* **target** ([*object*](https://docs.python.org/3/builtins/functions.html#object)) – the object on which the glom will operate.
* **spec** ([*object*](https://docs.python.org/3/builtins/functions.html#object)) – Specification of the output object in the form
of a dict, list, tuple, string, other glom construct, or
any composition of these.
* **default** ([*object*](https://docs.python.org/3/builtins/functions.html#object)) – An optional default to return in the case
an exception, specified by *skip_exc*, is raised.
* **skip_exc** ([*Exception*](https://docs.python.org/3/builtins/exceptions.html#Exception)) – An optional exception or tuple of
exceptions to ignore and return *default* (None if
omitted). If *skip_exc* and *default* are both not set,
glom raises errors through.
* **scope** ([*dict*](https://docs.python.org/3/builtins/stdtypes.html#dict)) – Additional data that can be accessed
via S inside the glom-spec.
It’s a small API with big functionality, and glom’s power is
only surpassed by its intuitiveness. Give it a whirl!
### py2store.utils.glom.is_iterable(x)
Similar in nature to [`callable()`](https://docs.python.org/3/builtins/functions.html#callable), `is_iterable` returns
`True` if an object is iterable, `False` if not.
```pycon
>>> is_iterable([])
True
>>> is_iterable(1)
False
```
### py2store.utils.glom.make_sentinel(name='_MISSING', var_name=None)
Creates and returns a new **instance** of a new class, suitable for
usage as a “sentinel”, a kind of singleton often used to indicate
a value is missing when `None` is a valid input.
* **Parameters:**
* **name** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – Name of the Sentinel
* **var_name** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – Set this name to the name of the variable in
its respective module enable pickleability.
```pycon
>>> make_sentinel(var_name='_MISSING')
_MISSING
```
The most common use cases here in boltons are as default values
for optional function arguments, partly because of its
less-confusing appearance in automatically generated
documentation. Sentinels also function well as placeholders in queues
and linked lists.
#### NOTE
By design, additional calls to `make_sentinel` with the same
values will not produce equivalent objects.
```pycon
>>> make_sentinel('TEST') == make_sentinel('TEST')
False
>>> type(make_sentinel('TEST')) == type(make_sentinel('TEST'))
False
```
### py2store.utils.glom.register(target_type, \*\*kwargs)
Register *target_type* so `glom()` will
know how to handle instances of that type as targets.
* **Parameters:**
* **target_type** ([*type*](https://docs.python.org/3/builtins/functions.html#type)) – A type expected to appear in a glom()
call target
* **get** (*callable*) – A function which takes a target object and
a name, acting as a default accessor. Defaults to
[`getattr()`](https://docs.python.org/3/builtins/functions.html#getattr).
* **iterate** (*callable*) – A function which takes a target object
and returns an iterator. Defaults to [`iter()`](https://docs.python.org/3/builtins/functions.html#iter) if
*target_type* appears to be iterable.
* **exact** ([*bool*](https://docs.python.org/3/builtins/functions.html#bool)) – Whether or not to match instances of subtypes
of *target_type*.
#### NOTE
The module-level [`register()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.register) function affects the
module-level [`glom()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.glom) function’s behavior. If this
global effect is undesirable for your application, or
you’re implementing a library, consider instantiating a
[`Glommer`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Glommer) instance, and using the
[`register()`](_autosummary/py2store.utils.glom.html.md#py2store.utils.glom.Glommer.register) and `Glommer.glom()`
methods instead.
### py2store.utils.glom.register_op(op_name, \*\*kwargs)
For extension authors needing to add operations beyond the builtin
‘get’ and ‘iterate’ to the default scope. See TargetRegistry for more details.
# _autosummary/py2store.utils.html.md
# py2store.utils
general utils
### Modules
| [`affine_conversion`](_autosummary/py2store.utils.affine_conversion.html.md#module-py2store.utils.affine_conversion) | utils to carry out affine transformations (of indices) |
|--------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`appendable`](_autosummary/py2store.utils.appendable.html.md#module-py2store.utils.appendable) | utils to make add append and extend functionality to KV stores |
| [`attr_dict`](_autosummary/py2store.utils.attr_dict.html.md#module-py2store.utils.attr_dict) | a data object layer for object attributes |
| [`cache_descriptors`](_autosummary/py2store.utils.cache_descriptors.html.md#module-py2store.utils.cache_descriptors) | descriptors to cache data |
| [`cumul_aggreg_write`](_autosummary/py2store.utils.cumul_aggreg_write.html.md#module-py2store.utils.cumul_aggreg_write) | utils for bulk writing -- accumulate, aggregate and write when some condition is met |
| [`explicit`](_autosummary/py2store.utils.explicit.html.md#module-py2store.utils.explicit) | utils to make stores based on a the input data itself |
| [`glom`](_autosummary/py2store.utils.glom.html.md#module-py2store.utils.glom) | *glom is a util to extract stuff from nested structures.* It's one of those excellent utils that I've written many times, but never got quite right. |
| [`mappify`](_autosummary/py2store.utils.mappify.html.md#module-py2store.utils.mappify) | Utils to wrap any object into a mapping interface |
| [`mg_selectors`](_autosummary/py2store.utils.mg_selectors.html.md#module-py2store.utils.mg_selectors) | Selectors that use the mongo-query interface |
| [`mongoquery`](_autosummary/py2store.utils.mongoquery.html.md#module-py2store.utils.mongoquery) | Transform mongo-like selector dicts (filters) into boolean functions that implement the condition |
| [`signatures`](_autosummary/py2store.utils.signatures.html.md#module-py2store.utils.signatures) | Deprecated: Forwards to py2store.signatures |
| [`timeseries_caching`](_autosummary/py2store.utils.timeseries_caching.html.md#module-py2store.utils.timeseries_caching) | Tools to cache time-series data. |
| [`uri_utils`](_autosummary/py2store.utils.uri_utils.html.md#module-py2store.utils.uri_utils) | utils to work with URIs |
# _autosummary/py2store.utils.mappify.html.md
# py2store.utils.mappify
Utils to wrap any object into a mapping interface
### Functions
| `bracket_getter`(obj, k) | |
|------------------------------------------------|----|
| `dot_str_key_iterator`(p) | |
| `simple_glom`(target, spec[, node_types, ...]) | |
### Classes
| [`LeafMappify`](_autosummary/py2store.utils.mappify.html.md#py2store.utils.mappify.LeafMappify)(target[, node_types, ...]) | A dict-like interface to glom. |
|-------------------------------------------------------------------------------------------------|----------------------------------|
| [`Mappify`](_autosummary/py2store.utils.mappify.html.md#py2store.utils.mappify.Mappify)(target[, node_types, key_concat, ...]) | |
### *class* py2store.utils.mappify.LeafMappify(target, node_types=(, ), key_concat=>, names_of_literals=(), \*\*kwargs)
Bases: [`Mappify`](_autosummary/py2store.utils.mappify.html.md#py2store.utils.mappify.Mappify)
A dict-like interface to glom. Here, only leaf keys are taken into account.
```pycon
>>> d = {
... 'a': 'simple',
... 'b': {'is': 'nested'},
... 'c': {'is': 'nested', 'and': 'has', 'a': [1, 2, 3]}
... }
>>> g = LeafMappify(d)
>>>
>>> assert list(g) == ['a', 'b.is', 'c.is', 'c.and', 'c.a']
>>> assert g['a'] == 'simple'
>>> assert g['b.is'] == 'nested'
>>> assert g['c.a'] == [1, 2, 3]
>>>
>>> for k, v in g.items():
... print(f"{k}: {v}")
...
a: simple
b.is: nested
c.is: nested
c.and: has
c.a: [1, 2, 3]
```
### *class* py2store.utils.mappify.Mappify(target, node_types=(, ), key_concat=>, names_of_literals=(), \*\*kwargs)
Bases: `KvReader`
```pycon
>>> d = {
... 'a': 'simple',
... 'b': {'is': 'nested'},
... 'c': {'is': 'nested', 'and': 'has', 'a': [1, 2, 3]}
... }
>>> g = Mappify(d)
>>>
>>> assert list(g) == ['a', 'b.is', 'b', 'c.is', 'c.and', 'c.a', 'c']
>>> assert g['a'] == 'simple'
>>> assert g['b.is'] == 'nested'
>>> assert g['c.a'] == [1, 2, 3]
>>>
>>> for k, v in g.items():
... print(f"{k}: {v}")
...
a: simple
b.is: nested
b: {'is': 'nested'}
c.is: nested
c.and: has
c.a: [1, 2, 3]
c: {'is': 'nested', 'and': 'has', 'a': [1, 2, 3]}
```
# _autosummary/py2store.utils.mg_selectors.html.md
# py2store.utils.mg_selectors
Selectors that use the mongo-query interface
# _autosummary/py2store.utils.mongoquery.html.md
# py2store.utils.mongoquery
Transform mongo-like selector dicts (filters) into boolean functions that implement the condition
Modified from mongoquery ([https://github.com/kapouille/mongoquery](https://github.com/kapouille/mongoquery))
mongoquery provides a straightforward API to match Python objects against
MongoDB Query Language queries.
### Functions
| [`is_non_string_sequence`](_autosummary/py2store.utils.mongoquery.html.md#py2store.utils.mongoquery.is_non_string_sequence)(entry) | Returns True if entry is a Python sequence iterable, and not a string |
|----------------------------------------------------------------------------------|-------------------------------------------------------------------------|
### Classes
| [`Query`](_autosummary/py2store.utils.mongoquery.html.md#py2store.utils.mongoquery.Query)(definition) | The Query class is used to match an object against a MongoDB-like query |
|----------------------------------------------------------------------|---------------------------------------------------------------------------|
### Exceptions
| [`QueryError`](_autosummary/py2store.utils.mongoquery.html.md#py2store.utils.mongoquery.QueryError) | Query error exception |
|---------------------------------------------------------------|-------------------------|
### *class* py2store.utils.mongoquery.Query(definition)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The Query class is used to match an object against a MongoDB-like query
#### match(entry)
Matches the entry object against the query specified on instanciation
### *exception* py2store.utils.mongoquery.QueryError
Bases: [`Exception`](https://docs.python.org/3/builtins/exceptions.html#Exception)
Query error exception
### py2store.utils.mongoquery.is_non_string_sequence(entry)
Returns True if entry is a Python sequence iterable, and not a string
# _autosummary/py2store.utils.signatures.html.md
# py2store.utils.signatures
Deprecated: Forwards to py2store.signatures
# _autosummary/py2store.utils.timeseries_caching.html.md
# py2store.utils.timeseries_caching
Tools to cache time-series data.
### Classes
| [`RegularTimeseriesCache`](_autosummary/py2store.utils.timeseries_caching.html.md#py2store.utils.timeseries_caching.RegularTimeseriesCache)([data_rate, ...]) | A type that pretends to be a (possibly very large) list, but where contents of the list are populated as they are needed. |
|---------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
### *class* py2store.utils.timeseries_caching.RegularTimeseriesCache(data_rate=1, time_rate=1, maxlen=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A type that pretends to be a (possibly very large) list, but where contents of the list are populated as they are
needed. Further, the indexing of the list can be overwritten for the convenience of the user.
The canonical application is where we have segments of continuous waveform indexed by utc microseconds timestamps.
It is convenient to be able to read segments of this waveform as if it was one big waveform (handling the
discontinuities gracefully), and have the choice of using (relative or absolute) integer indices or utc indices.
# _autosummary/py2store.utils.uri_utils.html.md
# py2store.utils.uri_utils
utils to work with URIs
### Functions
| [`build_uri`](_autosummary/py2store.utils.uri_utils.html.md#py2store.utils.uri_utils.build_uri)(scheme[, database, username, ...]) | Reverse of `parse_uri` function. |
|-------------------------------------------------------------------------------------------------|---------------------------------------------|
| `mk_str_making_func`(str_format[, ...]) | |
| [`parse_uri`](_autosummary/py2store.utils.uri_utils.html.md#py2store.utils.uri_utils.parse_uri)(uri) | Parses DB URI string into a dict of params. |
### py2store.utils.uri_utils.build_uri(scheme, database='', username=None, password=None, host='localhost', port=None)
Reverse of `parse_uri` function.
Builds a URI string from provided params.
### py2store.utils.uri_utils.parse_uri(uri)
Parses DB URI string into a dict of params.
* **Parameters:**
**uri** – string formatted as: “scheme://username:password@host:port/database”
* **Returns:**
a dict with these params parsed.
# about-this-build.html.md
# About this build
This documentation was built on **2026-09-15 11:34 UTC** from commit 03fbcfc on branch master, for **py2store 0.1.23** (from setup.cfg).
#### NOTE
Nothing suggests a mismatch: the tree was clean at the commit above, and the documented version is the one on PyPI.
## Source
| | |
|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
| Commit | 03fbcfc37164a618f0b92b7ac3c83b7157747efa |
| Branch | master |
| Tags at this commit | 0.1.23 |
| Working tree | clean |
| Remote | https://github.com/i2mint/py2store |
## Continuous integration
| | |
|--------------|--------------------------------------------------------------------------------------------|
| Repository | i2mint/py2store |
| Run | 34963902384 |
| Ref | refs/heads/master |
| Event commit | db92faff58d7acbc060f41aa54c127f37ee13ac8 (in the history of the built commit) |
## Tools
| | |
|----------|---------|
| epythet | 0.2.11 |
| Sphinx | 9.1.0 |
| docutils | 0.22.4 |
| Python | 3.12.14 |
## Configuration as resolved
| | |
|---------------|------------------------------------------------------------------|
| theme | auto (Sphinx theme furo) |
| accent | #87356b |
| api_generator | autosummary |
| ignore | tests/, scrap/, examples/ |
| agent_outputs | true |
| aggregates | md |
| ai_artifacts | true |
## Package on PyPI
Latest release: 0.1.23, the same as the documented version.
## Reproduce
```bash
git clone https://github.com/i2mint/py2store && cd py2store
git checkout 03fbcfc37164a618f0b92b7ac3c83b7157747efa
pip install "epythet==0.2.11"
epythet quickstart . --ignore tests/ scrap/ examples/
```
The same data, for machines: build_info.json (schema version 1).
# api.html.md
# API reference
| [`py2store`](_autosummary/py2store.html.md#module-py2store) | py2store: tools to create simple and consistent interfaces to complicated and varied data sources. |
|-----------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|