Deduplicate Items Using Map Objects

The Remove Duplicates node works for small datasets, but slows significantly beyond a few thousand items. A Code node with a Map object is O(n) and handles 100k+ items efficiently. You also get full control over which fields define uniqueness.

Real-world example: Merging contacts from three different CRMs where the same person may appear with slightly different email capitalization or trailing whitespace.

// Mode: Run Once for All Items
const seen = new Map();
const duplicates = [];

for (const item of $input.all()) {
  // Build a composite dedup key: normalized email + last name
  const email = (item.json.email ?? '').toLowerCase().trim();
  const lastName = (item.json.lastName ?? '').toLowerCase().trim();
  const key = `${email}|${lastName}`;

  if (!key || key === '|') continue; // Skip items with no identifying info

  if (seen.has(key)) {
    // Keep the record with more complete data (more non-null fields)
    const existing = seen.get(key);
    const existingScore = Object.values(existing.json).filter(Boolean).length;
    const currentScore = Object.values(item.json).filter(Boolean).length;

    if (currentScore > existingScore) {
      duplicates.push(existing);
      seen.set(key, item);
    } else {
      duplicates.push(item);
    }
  } else {
    seen.set(key, item);
  }
}

// Return deduplicated items
// Optionally: log duplicates count for monitoring
const results = Array.from(seen.values());
results.push({
  json: {
    _metadata: true,
    totalInput: $input.all().length,
    uniqueOutput: seen.size,
    duplicatesRemoved: duplicates.length,
  }
});

return results;
```text
| Dataset Size | Remove Duplicates Node | Code Node (Map) |
|:------------|:----------------------|:----------------|
| 1,000 items | ~200ms | ~15ms |
| 10,000 items | ~4s | ~80ms |
| 100,000 items | Timeout risk | ~600ms |

**Related:** [Use Edit Fields in "Map Each" Mode for Simple Renames](../data-transformation/01-use-edit-fields-in-map-each-mode-for-simple-renames.md) | [Configure Payload Size and Binary Data Mode for Large Files](../performance-and-large-files/01-configure-payload-size-and-binary-data-mode-for-large-files.md)