Process Large Datasets in Batches With Split In Batches

Attempting to process 50,000 records in a single pass will exhaust memory, trigger API rate limits, and create execution payloads too large to store. The Split In Batches node breaks large datasets into manageable chunks.

Real-world example: Importing 50,000 contacts from a CSV export into HubSpot, which has a rate limit of 100 requests per 10 seconds.

Workflow structure:

[Read CSV File] -> [Split In Batches (batch size: 50)] -> [HubSpot Create Contact]
                         ^                                        |
                         |________________________________________|
                                    (loop back)
```text
Configuration for the Split In Batches node:

| Setting | Value | Reason |
|:--------|:------|:-------|
| Batch Size | 50 | Stay well under HubSpot's 100/10s rate limit |
| Options > Reset | false | Continue from where we left off on errors |

```javascript
// Optional: Add a Code node after HubSpot to add a delay between batches
// This prevents hitting rate limits on APIs with sliding windows
const batchIndex = $input.first().json.$batchIndex ?? 0;

// Wait 1 second every 5 batches (250 records)
if (batchIndex > 0 && batchIndex % 5 === 0) {
  await new Promise(resolve => setTimeout(resolve, 1000));
}

return $input.all();
```text
> **Tip: Monitor Batch Progress**
>
> Add a Set node inside the loop that writes the current batch number to static workflow data using `$workflow.staticData.lastBatch = batchIndex`. If the workflow fails mid-import, you can check this value and resume from the correct batch.

**Related:** [Flatten Deeply Nested API Responses](../code-node-mastery/01-flatten-deeply-nested-api-responses.md) | [Use Docker Compose with Health Checks for n8n and PostgreSQL](../self-hosting-operations/01-use-docker-compose-with-health-checks-for-n8n-and-postgresql.md)