Use the Text Splitter Sub-Node for Chunking Documents Before Embedding

Embedding models have token limits (typically 8,192 tokens for OpenAI's models). Large documents must be split into chunks before embedding. The Text Splitter sub-node handles this with configurable chunk size and overlap. The overlap ensures that concepts spanning a chunk boundary are captured in at least one chunk.

Real-world example: You are building a RAG pipeline to index a 50-page employee handbook. Each page averages 800 tokens. You need to chunk it for embedding and storage.

Text Splitter sub-node configuration:

Setting	Value
Mode	Recursive Character Splitter
Chunk Size	500 (tokens)
Chunk Overlap	50 (tokens)
Separators	\n\n, \n, . ,

How chunking works visually:

Original document (1,500 tokens):
[===============================================================]

Chunk 1 (500 tokens):  [===================]
Chunk 2 (500 tokens):            [===================]     ← 50 token overlap
Chunk 3 (500 tokens):                      [===================]
                                  ↑ overlap ↑
```text
Practical ingestion workflow:

```text
[Read PDF] → [Extract Text] → [Text Splitter: 500/50] → [Embeddings: OpenAI] → [Vector Store: Insert]
```text
Code to add metadata to each chunk before embedding:

```javascript
const items = $input.all();

return items.map((item, index) => ({
  json: {
    content: item.json.text,
    metadata: {
      source: 'employee_handbook_v3.pdf',
      chunk_index: index,
      total_chunks: items.length,
      section: item.json.loc?.pageNumber
        ? `Page ${item.json.loc.pageNumber}`
        : `Chunk ${index + 1}`
    }
  }
}));
```text
> **Note: Chunk Size Guidelines**
>
> | Document Type          | Chunk Size | Overlap | Rationale                              |
> |------------------------|------------|---------|----------------------------------------|
> | Technical docs         | 500 tokens | 50      | Dense information, needs context        |
> | Conversational content | 300 tokens | 30      | Shorter natural units                   |
> | Legal documents        | 800 tokens | 100     | Long clauses need more context          |
> | Code files             | 400 tokens | 80      | Functions often span boundaries         |

Proper chunking directly determines retrieval quality. Too large and you waste context window on irrelevant text; too small and you lose the context needed to answer questions.

**Related:** [Use Manual Trigger During Development Instead of Webhook or Schedule](../api-cost-optimization/01-use-manual-trigger-during-development-instead-of-webhook-or-schedule.md) | [Flatten Deeply Nested API Responses](../code-node-mastery/01-flatten-deeply-nested-api-responses.md)