You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Deepanshu Lodhi <De...@inventia.in> on 2026/09/07 07:53:14 UTC

NiFi Batch-Based Duplicate Handling with PutDatabaseRecord in PostgreSQL

Hi everyone,
I have a requirement where a FlowFile contains 10K records, of which around 10 records may already exist in the target PostgreSQL table. Using PutDatabaseRecord, is there a way to insert only the non-existing records while routing the duplicates separately to another PutDatabaseRecord for logging into a duplicate/replica table?
The data volume is very high, so record-level splitting is not feasible. The solution must be NiFi-based only, without Stored Procedures, and duplicate detection should ideally be batch/set-based rather than one DB lookup per record.
What would be the recommended approach/processors for this scenario?
Regards
-Deepanshu