Use Compare Datasets to Detect New, Updated, and Deleted Records

The Compare Datasets node compares two snapshots of data and sorts records into new, updated, and unchanged or deleted buckets. It is the foundation of incremental sync workflows that avoid reprocessing an entire dataset on every run: match records on a key field, compare selected fields, and route each output branch to the right action such as a welcome message, a CRM update, or a churn log.

What does the Compare Datasets node do?

The Compare Datasets node compares two snapshots of data and categorizes records into three buckets: new, updated, and unchanged (or deleted). This is the foundation of incremental sync workflows that avoid reprocessing the entire dataset on every run.

How do you configure Compare Datasets for incremental sync?

Real-world example: Every hour, you pull the subscriber list from Mailchimp. You need to detect which subscribers are new (to send a welcome Slack message), which changed their email preferences (to update your CRM), and which unsubscribed (to log the churn).

Workflow structure:

[Get Current Subscribers] ──────► [Compare Datasets] ──► [New] ──► Slack Welcome
         (Mailchimp)                     ▲               [Updated] ──► CRM Update
                                         │               [Deleted] ──► Churn Log
[Get Previous Snapshot] ────────────────┘
         (Database)

Configure the Compare Datasets node:

Setting	Value
Input A (current data)	Connected to Mailchimp node
Input B (previous data)	Connected to Database node
Fields to Match	email
Fields to Compare	status, preferences

The node produces three output branches:

// Output 1 — New items (in A but not in B):
{ "email": "newuser@example.com", "status": "subscribed", "name": "New User" }

// Output 2 — Updated items (in both, but fields differ):
{
  "email": "existing@example.com",
  "status": "unsubscribed",  // changed from "subscribed"
  "name": "Existing User"
}

// Output 3 — Unchanged items (in both, fields identical):
{ "email": "stable@example.com", "status": "subscribed", "name": "Stable User" }

This pattern reduces API calls, prevents duplicate notifications, and gives you audit-ready change tracking.

How do you keep the snapshot current?

Tip: Save the Snapshot

After comparison, upsert the current dataset into your snapshot table so the next run compares against fresh data. Use ON CONFLICT (email) DO UPDATE in Postgres or equivalent.

Related: Flatten Deeply Nested API Responses · Use the HTTP Request Node as a Universal Connector