GetMongo question

[nadine giampapa <nm...@gmail.com>]

I am using the getMongo processor but every time it runs, it is returning
results from the beginning of time. I have my processor schedule to run
every ten minutes. I would like it to return only the new results since the
last time the processor ran. Is there a way to do that?

Re: GetMongo question

[Simon Ball <sb...@hortonworks.com>]

Not at present, unless you customize the query attribute yourself based on the results of the previous query, which is possible but a little fiddly.

A better solution would be to add an incremental processor for Mongo, similar to QueryDatabaseTable, which uses the NiFi State storage mechanism to maintain latest timestamps. Given the way Mongo ObjectIDs embed a timestamp, this could be done quite neatly.

Would you be interested on working on contributing something like that?

The first step would be to put in an enhancement ticket at https://issues.apache.org/jira/browse/NIFI

Simon

On Apr 27, 2016, at 10:15 PM, nadine giampapa <nm...@gmail.com>> wrote:

I am using the getMongo processor but every time it runs, it is returning results from the beginning of time. I have my processor schedule to run every ten minutes. I would like it to return only the new results since the last time the processor ran. Is there a way to do that?