Deduplicate Items Using Map Objects

To deduplicate large item sets in n8n, use a Code node with a JavaScript Map instead of the Remove Duplicates node. The Map runs in O(n) time, handles 100k-plus items, and lets you define a composite key from any fields. On a collision you keep the record with the most complete data and track how many duplicates were removed.

Why deduplicate with a Map instead of the Remove Duplicates node?

The Remove Duplicates node works for small datasets, but slows significantly beyond a few thousand items. A Code node with a Map object is O(n) and handles 100k+ items efficiently. You also get full control over which fields define uniqueness.

Real-world example: Merging contacts from three different CRMs where the same person may appear with slightly different email capitalization or trailing whitespace.

How do you deduplicate items with a Map in a Code node?

// Mode: Run Once for All Items
const seen = new Map();
const duplicates = [];

for (const item of $input.all()) {
  // Build a composite dedup key: normalized email + last name
  const email = (item.json.email ?? '').toLowerCase().trim();
  const lastName = (item.json.lastName ?? '').toLowerCase().trim();
  const key = `${email}|${lastName}`;

  if (!key || key === '|') continue; // Skip items with no identifying info

  if (seen.has(key)) {
    // Keep the record with more complete data (more non-null fields)
    const existing = seen.get(key);
    const existingScore = Object.values(existing.json).filter(Boolean).length;
    const currentScore = Object.values(item.json).filter(Boolean).length;

    if (currentScore > existingScore) {
      duplicates.push(existing);
      seen.set(key, item);
    } else {
      duplicates.push(item);
    }
  } else {
    seen.set(key, item);
  }
}

// Return deduplicated items
// Optionally: log duplicates count for monitoring
const results = Array.from(seen.values());
results.push({
  json: {
    _metadata: true,
    totalInput: $input.all().length,
    uniqueOutput: seen.size,
    duplicatesRemoved: duplicates.length,
  }
});

return results;

How does Map deduplication compare on performance?

Dataset Size	Remove Duplicates Node	Code Node (Map)
1,000 items	~200ms	~15ms
10,000 items	~4s	~80ms
100,000 items	Timeout risk	~600ms

Related: Use Edit Fields in "Map Each" Mode for Simple Renames · Configure Payload Size and Binary Data Mode for Large Files