Process Large Datasets in Batches With Split In Batches

Processing tens of thousands of records in a single pass exhausts memory, triggers API rate limits, and creates execution payloads too large to store. The Split In Batches node breaks a large dataset into fixed-size chunks and loops over them, so each iteration handles only a manageable slice -- for example, 50 records at a time to stay under an API's rate limit.

Real-world example: Importing 50,000 contacts from a CSV export into HubSpot, which has a rate limit of 100 requests per 10 seconds.

How does the Split In Batches node work?

Workflow structure:

[Read CSV File] -> [Split In Batches (batch size: 50)] -> [HubSpot Create Contact]
                         ^                                        |
                         |________________________________________|
                                    (loop back)

How do you configure Split In Batches to respect rate limits?

Configuration for the Split In Batches node:

Setting	Value	Reason
Batch Size	50	Stay well under HubSpot's 100/10s rate limit
Options > Reset	false	Continue from where we left off on errors

Optionally, add a Code node after the API call to insert a delay between batches, which prevents hitting rate limits on APIs with sliding windows:

// Optional: Add a Code node after HubSpot to add a delay between batches
// This prevents hitting rate limits on APIs with sliding windows
const batchIndex = $input.first().json.$batchIndex ?? 0;

// Wait 1 second every 5 batches (250 records)
if (batchIndex > 0 && batchIndex % 5 === 0) {
  await new Promise(resolve => setTimeout(resolve, 1000));
}

return $input.all();

Tip: Monitor Batch Progress

Add a Set node inside the loop that writes the current batch number to static workflow data using $workflow.staticData.lastBatch = batchIndex. If the workflow fails mid-import, you can check this value and resume from the correct batch.

Related: Flatten Deeply Nested API Responses · Use Docker Compose with Health Checks for n8n and PostgreSQL