chromadol.data_loaders

Data loaders for ChromaDB.

Functions

load_bytes(filepath)
load_text(filepath)
pdf_file_text(filepath, *[, page_break_delim])
test_file_loader()
url_to_contents(url[, content_extractor, params])

Classes

FileLoader([loader, prefix, suffix, max_workers])	A DataLoader that loads a list of text files from a list of URIs.
MappingLoader(mapping, *[, key_type, ...])

class chromadol.data_loaders.FileLoader(loader=<function load_text>, *, prefix='', suffix='', max_workers=4)

Bases: DataLoader[list[str | bytes | None]]

A DataLoader that loads a list of text files from a list of URIs.

By default, it loads text files from local files, given URIs that are full file paths. You can specify a prefix thought (usually used to specify a root directory), or a suffix (usually used to specify a file extension).

Further, you can specify a different loader, e.g. to load files from a remote URL, or to load binary files, or to load text from pdf files, or from S3, or a database, etc.

Example

>>> import chromadb
>>> rootdir = chromadb.__path__[0] + '/'
>>> file_loader_1 = FileLoader(prefix=rootdir)
>>> file_contents_1 = file_loader_1(['__init__.py', 'types.py'])
>>> len(file_contents_1)
2
>>> 'Embeddings' in file_contents_1[0]  # i.e. __init__.py contains the word 'Embeddings'
True
>>> 'from typing import' in file_contents_1[1]  # i.e. types.py contains the phrase 'from typing import'
True

class chromadol.data_loaders.MappingLoader(mapping, *, key_type=<class 'str'>, ingress=<function identity>, egress=<function identity>, max_workers=None)

Bases: DataLoader