vd.text#

Text preprocessing and chunking utilities for vd.

Provides functions to clean, normalize, and chunk text before adding to vector databases.

Functions

chunk_documents(documents[, chunk_size, ...])

Chunk multiple documents while preserving metadata.

chunk_text(text[, chunk_size, overlap, ...])

Chunk text into smaller pieces.

clean_text(text, *[, lowercase, ...])

Clean and normalize text.

extract_metadata(text, *[, extract_title, ...])

Extract metadata from text.

normalize_whitespace(text)

Normalize whitespace in text.

truncate_text(text, max_length, *[, suffix])

Truncate text to maximum length.
vd.text.chunk_documents(documents, chunk_size=500, *, overlap=50, strategy='chars', id_template='{doc_id}_chunk_{chunk_num}', preserve_metadata=True)[source]#

Chunk multiple documents while preserving metadata.

Parameters:

  • documents (Iterator[tuple[str, str | dict]]) – Iterator of (doc_id, text) or (doc_id, text, metadata) tuples

  • chunk_size (int) – Size of each chunk

  • overlap (int) – Overlap between chunks

  • strategy (str) – Chunking strategy (see chunk_text)

  • id_template (str) – Template for chunk IDs. Can use {doc_id} and {chunk_num}

  • preserve_metadata (bool) – Whether to copy metadata to all chunks

Yields:

tuple – (chunk_id, chunk_text, metadata) tuples

Return type:

Iterator[tuple[str, str, dict]]

Examples

>>> docs = [('doc1', 'Long text...', {'author': 'Alice'})]
>>> chunks = list(chunk_documents(docs, chunk_size=20))
>>> len(chunks) >= 1
True
vd.text.chunk_text(text, chunk_size=500, *, overlap=50, strategy='chars', preserve_sentences=True)[source]#

Chunk text into smaller pieces.

Parameters:

  • text (str) – Text to chunk

  • chunk_size (int) – Target size of each chunk (in characters or tokens depending on strategy)

  • overlap (int) – Number of characters/tokens to overlap between chunks

  • strategy (str) –

    Chunking strategy:

    • ’chars’: Character-based chunking

    • ’words’: Word-based chunking

    • ’sentences’: Sentence-based chunking

    • ’paragraphs’: Paragraph-based chunking

  • preserve_sentences (bool) – Try to avoid breaking sentences when using chars/words strategy

Returns:

List of text chunks

Return type:

list[str]

Examples

>>> text = "This is sentence one. This is sentence two. This is sentence three."
>>> chunks = chunk_text(text, chunk_size=30, strategy='chars')
>>> len(chunks) >= 2
True

>>> chunks = chunk_text(text, strategy='sentences')
>>> len(chunks)
3
vd.text.clean_text(text, *, lowercase=False, remove_extra_whitespace=True, remove_urls=False, remove_emails=False, remove_numbers=False, remove_punctuation=False)[source]#

Clean and normalize text.

Parameters:

  • text (str) – Text to clean

  • lowercase (bool) – Convert to lowercase

  • remove_extra_whitespace (bool) – Collapse multiple spaces/newlines

  • remove_urls (bool) – Remove URLs

  • remove_emails (bool) – Remove email addresses

  • remove_numbers (bool) – Remove numbers

  • remove_punctuation (bool) – Remove punctuation

Returns:

Cleaned text

Return type:

str

Examples

>>> text = "Hello   World!  Visit https://example.com"
>>> clean_text(text, remove_urls=True)
'Hello World! Visit'
>>> clean_text(text, lowercase=True, remove_punctuation=True)
'hello world visit httpsexamplecom'
vd.text.extract_metadata(text, *, extract_title=True, extract_length=True, extract_word_count=True, extract_language=False)[source]#

Extract metadata from text.

Parameters:

  • text (str) – Text to analyze

  • extract_title (bool) – Extract first line as title

  • extract_length (bool) – Add text length

  • extract_word_count (bool) – Add word count

  • extract_language (bool) – Detect language (requires langdetect)

Returns:

Extracted metadata

Return type:

dict[str, Any]

Examples

>>> text = "My Title\n\nThis is the content."
>>> meta = extract_metadata(text)
>>> meta['title']
'My Title'
>>> meta['char_count']
30
vd.text.normalize_whitespace(text)[source]#

Normalize whitespace in text.

Replaces tabs, multiple spaces, and multiple newlines with single versions.

Parameters:

text (str) – Text to normalize

Returns:

Normalized text

Return type:

str

Examples

>>> normalize_whitespace("Hello\t\tWorld  \n\n\nTest")
'Hello World \nTest'
vd.text.truncate_text(text, max_length, *, suffix='...')[source]#

Truncate text to maximum length.

Parameters:

  • text (str) – Text to truncate

  • max_length (int) – Maximum length

  • suffix (str) – Suffix to add to truncated text

Returns:

Truncated text

Return type:

str

Examples

>>> truncate_text("This is a long text", 10)
'This is...'