py2store.slib.s_zipfile¶
a data object layer for zipfile
-
class
py2store.slib.s_zipfile.
FileStreamsOfZip
(zip_file, prefix='', open_kws=None)[source]¶ Like FilesOfZip, but object returns are file streams instead. So you use it like this:
``` z = FileStreamsOfZip(rootdir) with z[relpath] as fp:
… # do stuff with fp, like fp.readlines() or such…
-
class
py2store.slib.s_zipfile.
FlatZipFilesReader
(rootdir, subpath='.+\\.zip', pattern_for_field=None, max_levels=0, zip_reader=<class 'py2store.slib.s_zipfile.ZipReader'>, **zip_reader_kwargs)[source]¶ Read the union of the contents of multiple zip files. A local file reader whose keys are the zip filepaths of the rootdir and values are corresponding ZipReaders.
-
py2store.slib.s_zipfile.
ZipFileReader
¶
-
class
py2store.slib.s_zipfile.
ZipFileStreamsReader
(rootdir, subpath='.+\\.zip', pattern_for_field=None, max_levels=0, *, zip_reader=<class 'py2store.slib.s_zipfile.FileStreamsOfZip'>, **zip_reader_kwargs)¶ Like ZipFilesReader, but objects returned are file streams instead.
-
class
py2store.slib.s_zipfile.
ZipFilesReader
(rootdir, subpath='.+\\.zip', pattern_for_field=None, max_levels=0, zip_reader=<class 'py2store.slib.s_zipfile.ZipReader'>, **zip_reader_kwargs)[source]¶ A local file reader whose keys are the zip filepaths of the rootdir and values are corresponding ZipReaders.
-
class
py2store.slib.s_zipfile.
ZipFilesReaderAndBytesWriter
(rootdir, subpath='.+\\.zip', pattern_for_field=None, max_levels=0, zip_reader=<class 'py2store.slib.s_zipfile.ZipReader'>, **zip_reader_kwargs)[source]¶ Like ZipFilesReader, but the ability to write bytes (assumed to be valid bytes of the zip format) to a key
-
class
py2store.slib.s_zipfile.
ZipReader
(zip_file, prefix='', open_kws=None, file_info_filt=None)[source]¶ A KvReader to read the contents of a zip file. Provides a KV perspective of https://docs.python.org/3/library/zipfile.html
ZipReader
has two value categories: Directories and Files. Both categories are distinguishable by the keys, through the “ends with slash” convention.When a file, the value return is bytes, as usual.
- When a directory, the value returned is a
ZipReader
itself, with all params the same, except for theprefix
which serves to specify the subfolder (that is, ``prefix` acts as a filter).
Note: If you get data zipped by a mac, you might get some junk along with it. Namely __MACOSX folders .DS_Store files. I won’t rant about it, since others have. But you might find it useful to remove them from view. One choice is to use py2store.trans.filt_iter to get a filtered view of the zips contents. In most cases, this should do the job:
` # applied to store instance or class: store = filt_iter(filt=lambda x: not x.startswith('__MACOSX') and '.DS_Store' not in x)(store) `
Another option is just to remove these from the zip file once and for all. In unix-like systems:
` zip -d filename.zip __MACOSX/\* zip -d filename.zip \*/.DS_Store `
Examples
# >>> s = ZipReader(‘/path/to/some_zip_file.zip’) # >>> len(s) # 53432 # >>> list(s)[:3] # the first 3 elements (well… their keys) # [‘odir/’, ‘odir/app/’, ‘odir/app/data/’] # >>> list(s)[-3:] # the last 3 elements (well… their keys) # [‘odir/app/data/audio/d/1574287049078391/m/Ctor.json’, # ‘odir/app/data/audio/d/1574287049078391/m/intensity.json’, # ‘odir/app/data/run/status.json’] # >>> # getting a file (note that by default, you get bytes, so need to decode) # >>> s[‘odir/app/data/run/status.json’].decode() # b’{“test_phase_number”: 9, “test_phase”: “TestActions.IGNORE_TEST”, “session_id”: 0}’ # >>> # when you ask for the contents for a key that’s a directory, # >>> # you get a ZipReader filtered for that prefix: # >>> s[‘odir/app/data/audio/’] # ZipReader(‘/path/to/some_zip_file.zip’, ‘odir/app/data/audio/’, {}, <function take_everything at 0x1538999e0>) # >>> # Often, you only want files (not directories) # >>> # You can filter directories out using the file_info_filt argument # >>> s = ZipReader(‘/path/to/some_zip_file.zip’, file_info_filt=ZipReader.FILES_ONLY) # >>> len(s) # compare to the 53432 above, that contained dirs too # 53280 # >>> list(s)[:3] # first 3 keys are all files now # [‘odir/app/data/plc/d/1574304926795633/d/1574305026895702’, # ‘odir/app/data/plc/d/1574304926795633/d/1574305276853053’, # ‘odir/app/data/plc/d/1574304926795633/d/1574305159343326’] # >>> # >>> # ZipReader.FILES_ONLY and ZipReader.DIRS_ONLY are just convenience filt functions # >>> # Really, you can provide any custom one yourself. # >>> # This filter function should take a ZipInfo object, and return True or False. # >>> # (https://docs.python.org/3/library/zipfile.html#zipfile.ZipInfo) # >>> # >>> import re # >>> p = re.compile(‘audio.*.json$’) # >>> my_filt_func = lambda fileinfo: bool(p.search(fileinfo.filename)) # >>> s = ZipReader(‘/Users/twhalen/Downloads/2019_11_21.zip’, file_info_filt=my_filt_func) # >>> len(s) # 48 # >>> list(s)[:3] # [‘odir/app/data/audio/d/1574333557263758/m/Ctor.json’, # ‘odir/app/data/audio/d/1574333557263758/m/intensity.json’, # ‘odir/app/data/audio/d/1574288084739961/m/Ctor.json’]
- When a directory, the value returned is a
-
class
py2store.slib.s_zipfile.
ZipStore
(zip_filepath, compression=8, allow_overwrites=True, pwd=None)[source]¶ Zip read and writing. When you want to read zips, there’s the FilesOfZip, ZipReader, or ZipFilesReader we know and love.
Sometimes though, you want to write to zips too. For this, we have ZipStore.
Since ZipStore can write to a zip, it’s read functionality is not going to assume static data, and cache things, as your favorite zip readers did. This, and the acrobatics need to disguise the weird zipfile into something more… key-value natural, makes for a not so efficient store, out of the box.
- I advise using one of the zip readers if all you need to do is read, or subclassing or
wrapping ZipStore with caching layers if it is appropriate to you.
-
py2store.slib.s_zipfile.
func_conjunction
(func1, func2)[source]¶ Returns a function that is equivalent to lambda x: func1(x) and func2(x)
-
py2store.slib.s_zipfile.
mk_flatzips_store
(dir_of_zips, zip_pair_path_preproc=<built-in function sorted>, mk_store=<class 'py2store.slib.s_zipfile.FlatZipFilesReader'>, **extra_mk_store_kwargs)[source]¶ A store so that you can work with a folder that has a bunch of zip files, as if they’ve all been extracted in the same folder. Note that zip_pair_path_preproc can be used to control how to resolve key conflicts (i.e. when you get two different zip files that have a same path in their contents). The last path encountered by zip_pair_path_preproc(zip_path_pairs) is the one that will be used, so one should make zip_pair_path_preproc act accordingly.