n8n Remove Duplicates Workflow: Dedupe Across Every Run

Duplicate records are the slow leak of data work. One contact becomes three across a year of imports, a webhook retry doubles an order, a nightly sync reprocesses the same rows it handled yesterday. An n8n remove duplicates workflow fixes the leak, but only if it dedupes across runs, not just within a single batch. That distinction is where most published examples fall short.

The official Remove Duplicates node docs explain the modes well. The popular templates, though, only do within-batch exact-match dedupe: clean the sheet in one pass and call it done. They skip cross-execution dedupe, which-record-to-keep logic, and anything resembling a near-match. This post covers all three.

What you can dedupe

The same node and patterns cover most of the real jobs:

• CRM contact lists that accumulate duplicates from multiple import sources
• Lead rows where the same person filled two forms
• Order or transaction tables hit by webhook retries
• Scraped data with repeated entries by URL or id
• Nightly syncs that shouldn't reprocess yesterday's rows
• Mailing lists deduped by normalized email before a send

The Deduplication Pipeline

Trigger → Read source → Normalize key → Sort (keeper first) → Remove Duplicates → Write back
                                                                      │
                                              within-input  OR  vs previous executions

The fork at the bottom is the decision that matters: are you cleaning one batch, or making sure you never process the same record twice across time?

1. Normalize the key first

Dedupe is only as good as the key. John@Example.com and john@example.com are the same person, but a byte comparison says they're different. Lowercase, trim, and strip formatting before you compare:

return items.map(({ json }) => ({
  json: { ...json, _key: String(json.email || '').trim().toLowerCase() },
}));

For phone numbers, strip everything but digits. For URLs, drop the trailing slash and query string. Skip this and your "dedupe" quietly keeps near-identical twins.

2. Decide which copy survives

Here's the opinionated take: the Remove Duplicates node drops extras, but it doesn't let you pick the keeper, so sort before you dedupe. If you want the most recent record to win, sort descending by date so it lands first; the node keeps the first occurrence. Want the most complete record? Compute a completeness score and sort on that. Letting the node pick arbitrarily is how you lose the good copy and keep the stub.

// keep the most recently updated copy
items.sort((a, b) => new Date(b.json.updated_at) - new Date(a.json.updated_at));
return items;

3. Within-input vs across-executions

This is the call the templates skip. Two modes, two jobs:

• Remove Items Repeated Within Current Input. This cleans the batch in front of the node. Use it for a one-time sheet cleanup or a freshly imported file.
• Remove Items Processed in Previous Executions. This compares each item against history the node persists, dropping anything it's seen before. Use it for recurring syncs so today's run ignores yesterday's rows.

The cross-execution mode stores up to 10,000 values by default at node or workflow scope. Node scope keeps each Remove Duplicates instance independent; workflow scope shares one history across several instances. Pick workflow scope when two different flows should respect the same "already processed" memory.

4. Handle near-matches when exact isn't enough

Exact-key dedupe misses Acme Inc vs Acme Inc. vs ACME, Inc. The built-in node won't catch those. When fuzzy matching matters, a Code node with a normalized comparison (lowercase, strip punctuation, optionally a Levenshtein distance) flags likely duplicates for review rather than auto-deleting them. Auto-merging fuzzy matches is risky; flag-and-review is the safer default.

5. Write back or merge

For a simple drop, write the survivors back to the source. To consolidate fields across duplicates (say, keep the newest email but the oldest signup date), merge in a Code node before removing, since the node itself doesn't merge.

Implementation patterns

Pattern 1: One-time sheet cleanup. Read the sheet, normalize the key, sort, Remove Duplicates in within-input mode, write the clean set back. The job most people start with.

Pattern 2: Idempotent recurring sync. A Schedule trigger pulls new rows; Remove Duplicates in previous-executions mode drops anything already handled, so re-runs are safe. This is the dedupe half of the webhook to database workflow.

Pattern 3: Dedupe-on-import. Fold the dedupe into the load step so the source table never accumulates twins in the first place. Pairs naturally with the CSV import workflow.

n8n nodes you'll use most

The Remove Duplicates node docs spell out the exact mode names and scope behaviour if you want the canonical reference.

Getting started

1. Choose the dedupe key and normalize it (lowercase, trim, strip formatting).
2. Decide the keeper rule and sort items so that copy is first.
3. Pick the mode: within-input for a cleanup, previous-executions for a recurring sync.
4. Set node or workflow scope based on whether other flows share the history.
5. For fuzzy needs, add a Code node that flags near-matches for review.
6. Write survivors back, or merge fields first if you need consolidation.
7. Run twice and confirm the second run drops everything from the first.

Getting the keeper logic and cross-run history right takes some fiddling. A template that bakes the dedupe into a validation gate gets you a clean, idempotent pipeline without the trial and error.

Deduping within a batch is table stakes; deduping across every run is what keeps the leak sealed for good. Normalize the key, sort the keeper first, and pick the mode that matches the job. That's the whole game. From here, harden the inputs with the data validation workflow and tour the rest of the data-ops catalog when your syncs need to stay clean on autopilot.