You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "David Handermann (Jira)" <ji...@apache.org> on 2024/01/20 02:55:00 UTC
[jira] [Updated] (NIFI-11129) Add PutMongoBulk Processor using Bulk Writes
[ https://issues.apache.org/jira/browse/NIFI-11129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Handermann updated NIFI-11129:
------------------------------------
Summary: Add PutMongoBulk Processor using Bulk Writes (was: Add PutMongoBulk processor - use the more efficient bulkWrite API)
> Add PutMongoBulk Processor using Bulk Writes
> --------------------------------------------
>
> Key: NIFI-11129
> URL: https://issues.apache.org/jira/browse/NIFI-11129
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: Sebastian Rothbucher
> Assignee: Sebastian Rothbucher
> Priority: Minor
> Fix For: 2.0.0-M2
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Problem to solve: PutMongo can only update (or upsert) one record at a time. For thousands of records, this is not efficient. The alternative PutMongoRecord assumes a lot about the documents written
> Proposed solution: add a PutMongoBulk processor that uses the bulkWrite API - bundle operations together into one (can be hundreds at a time, also by merging flowfiles). Can be any insert / replace / update / upsert with no assumption about how the documents look like. MongoDB processes all in one operation - resulting in way less load and way more efficient operation.
> What I can bring: I wrote such a processor about a year back (for earlier version); it did work in production flawlessly for months now. I did update it to 1.20-SNAPSHOT and re-tested; can submit a PR immediately after I get a ticket number here. Tried to be similar to PutMongo wherever poss; feedback is sure appreciated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)