Ever tried splitting text for your RAG pipeline and ended up with chunks that cut sentences in half? Or worse like chunks that lose all context between them?

Yeah, I've been there too. That's exactly why I built chunklet-py — a Python library that actually understands text structure.

This post hits only the highlights and doesn't cover everything. Visit the full documentation for everything else, including:

  • Custom sentence splitters for specialized languages
  • Custom document processors for unusual file formats
  • Custom tokenizers to match your LLM
  • The rich metadata you can get.
  • CLI flags for batch processing, parallel jobs, error handling, timeouts
  • Additional args like n_jobs, lang, show_progress, ...

⚠ Quick heads up!

This tutorial requires chunklet-py v2.2.0+ and uses APIs not available in earlier versions.

Upgrade to the latest version and see the documentation or What’s New for details.

The Problem with Dumb Splitting

Here's what usually happens:

```

The naive approach

chunks = [text[i:i+500] for i in range(0, len(text), 500)]
```
This works... until it doesn't:

  • Sentences cut mid-way
  • No context between chunks
  • Broken code if you're chunking source files

Solution: chunklet-py

A smart text and code chunking library that respects natural boundaries.

Features

  • 60+ languages supported:Auto-detects language and applies the right splitting rules. No more treating German the same as English.
  • Multiple constraint types:Mix and match:- max_sentencesgroup by sentences
  • max_tokensrespect LLM context limits
  • max_section_breakskeep Markdown headers together (headings- ##, horizontal rules- ---,- <details>tags)
  • max_linesfor code chunking

  • max_functionskeep functions together

  • Multiple file formats:PDF,PPTX, EML, DOCX, EPUB, HTML, Markdown, LaTeX, ODT, CSV, Excel, plain text — one library handles them all.

  • Rich metadata:Every chunk comes with source references, character spans, and structural info.
  • Composable constraints:Mix and match limits to get exactly the chunks you need.
  • Pluggable architecture:Swap in custom tokenizers, sentence splitters, or document processors.

Installation

pip install chunklet-py
For document support:

pip install chunklet-py[structured-document]
For code:

pip install chunklet-py[code]
For visualization:

pip install chunklet-py[visualization]

Code Examples

Core Imports

from chunklet import DocumentChunker # For PDFs, DOCX, and general text from chunklet import CodeChunker # For source code from chunklet import SentenceSplitter # For just sentences from chunklet import visualizer # Web-based visualizer

DocumentChunker API

Four methods cover most use cases:

| Method | Input | Return Type |
|---|---|---|
| chunk_text(text) | str | List[Chunk] |
| chunk_file(path) | Path or str | List[Chunk] |
| chunk_texts(list) | List[str] | Generator[Chunk] |
| chunk_files(list) | List[Path] | Generator[Chunk] |

DocumentChunker Example

```
chunker = DocumentChunker()

Feel free to mix and match these

chunks = chunker.chunk_text(
text,
max_sentences=3, # Stop after X sentences
max_tokens=500, # Don't blow up the LLM context
max_section_breaks=2, # Respect the Markdown headers
overlap_percent=20, # Give it some "memory" of the last chunk
offset=0 # Skip the first N sentences
)
```

CodeChunker Example

chunker = CodeChunker() chunks = chunker.chunk_text( code, max_lines=50, # Height limit max_tokens=512, # Width limit max_functions=1, # One function per chunk strict=True, # True: Crash on big blocks; False: Slice anyway include_comments=True, # True by default docstring_mode="all", # Options are: all, excluded, summary )

⚠ Token Counter Requirement

When using the max_tokens constraint, a token_counter function is essential. This function, which you provide, should accept a string and return an integer representing its token count. Failing to provide a token_counter will result in a MissingTokenCounterError.

You can also provide the token_counter directly to any chunking method. If provided in both the constructor and the method, the one in the method will be used.

SentenceSplitter (Just Sentences)

from chunklet import SentenceSplitter splitter = SentenceSplitter() sentences = splitter.split_text(text, lang="en") # You can also set it to "auto"
Handles tricky cases like "Dr." or "U.S.A." without breaking them up.

60+ languages are explicitly supported through dedicated libraries yasbd-lib covers 39+, Indic NLP Library covers 11, and Sentencex covers ~15, with some overlap), plus the Fallback Splitter handles any other language via Unicode rules (Supported Languages Documentation.

Output Object

Chunkers return Chunk objects (Box instances), so you use dot notation:

for chunk in chunks: print(chunk.content) # The actual text/code print(chunk.metadata) # Chunk metadata

Visualizer (Interactive Web UI)

Launch a web interface to experiment with chunking parameters:

chunklet visualize
Or programmatically:

from chunklet import visualizer v = visualizer.Visualizer(host="127.0.0.1", port=8000) v.serve() # Opens in your browser

CLI Examples

Prefer the terminal? chunklet-py ships with a full CLI

Here are some quick examples:

```

Basic text chunking

chunklet chunk "Your text here." --max-tokens 500

Chunk a file

chunklet chunk --source document.pdf --max-tokens 500 --metadata

Split text into sentences

chunklet split "Your text here." --lang en

Split a file into sentences

chunklet split --source my_file.txt --destination sentences.txt

Start the interactive visualizer

chunklet visualize

Code chunking

chunklet chunk --code --source my_script.py --max-functions 1

Batch processing a directory

chunklet chunk --doc --source ./my_docs --destination ./chunks --n-jobs 4

With error handling

chunklet chunk --doc --source ./my_docs --on-errors skip
```

How It Compares

While there are other chunking libraries available, Chunklet-py stands out for its unique combination of versatility, performance, and ease of use. Here's a quick look at how it compares to some of the alternatives:

| Library | Key Differentiator | Focus |
|---|---|---|
|
|
|
|
| Full LLM framework with basic splitters (e.g., RecursiveCharacterTextSplitter, Markdown, HTML, code splitters). Good for prototyping but basic for complex docs or multilingual needs. | Full Stack | |
| All-in-one pipeline (chunking + embeddings + vector DB). Uses tree-sitter for code. Multilingual. | Pipelines | |
| Text-only, fast semantic splitting. Built-in tiktoken/HuggingFace support. 85% faster than alternatives. | Text | |
| Code-specific, uses tree-sitter. Initially supports Python, JS, CSS only. | Code |

Chunklet-py is a specialized, drop-in replacement for the chunking step in any RAG pipeline. It handles text, documents, and code without heavy dependencies, while keeping your project lightweight.