Need help in getMongo

[Kavitha Alahari <Ka...@telus.com>]

Hello Team,

I am trying to pull the data from MongoDB and try to put in other Mongo DB. For this I used GetMongo and PutMongo processors. Unfortunately none of the query is working in getMongo. However it is working when no query is there. Could you please provide me a sample query to use in Get Mongo.

Also I have another requirement. Whenever data comes to a collection in MongoDB, the same data has to move into another collection of a different DB in the same server. Can we achieve this using Nifi.

Your help is really appreciated.

Regards
Kavitha Alahari

Re: Need help in getMongo

[Mike Thomsen <mi...@gmail.com>]

In fact, you probably don't even need one.

Consider this contrived example:

GetFile (load JSON)
  -> PutMongo (DB1)
     -> PutMongo (DB2)

If you have data coming in that needs to go to two Mongo databases, the
easiest way to handle it would be two PutMongos that act on the same
flowfile, either forking off from the same processor or chained together in
order of priority.

On Thu, Jun 29, 2017 at 12:49 PM, Pierre Villard <
pierre.villard.fr@gmail.com> wrote:

> Hi Kavitha,
>
> The query must be a JSON document, you have few examples in the test class
> GetMongoTest [1].
> Regarding your second question, there is no incremental fetch feature for
> the Mongo processors at the moment.
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/
> test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
>
> Pierre
>
> 2017-06-29 18:39 GMT+02:00 Kavitha Alahari <Ka...@telus.com>:
>
> > Hello Team,
> >
> > I am trying to pull the data from MongoDB and try to put in other Mongo
> > DB. For this I used GetMongo and PutMongo processors. Unfortunately none
> of
> > the query is working in getMongo. However it is working when no query is
> > there. Could you please provide me a sample query to use in Get Mongo.
> >
> > Also I have another requirement. Whenever data comes to a collection in
> > MongoDB, the same data has to move into another collection of a different
> > DB in the same server. Can we achieve this using Nifi.
> >
> > Your help is really appreciated.
> >
> > Regards
> > Kavitha Alahari
> >
>

Re: Need help in getMongo

[Pierre Villard <pi...@gmail.com>]

Hi Kavitha,

The query must be a JSON document, you have few examples in the test class
GetMongoTest [1].
Regarding your second question, there is no incremental fetch feature for
the Mongo processors at the moment.

[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java

Pierre

2017-06-29 18:39 GMT+02:00 Kavitha Alahari <Ka...@telus.com>:

> Hello Team,
>
> I am trying to pull the data from MongoDB and try to put in other Mongo
> DB. For this I used GetMongo and PutMongo processors. Unfortunately none of
> the query is working in getMongo. However it is working when no query is
> there. Could you please provide me a sample query to use in Get Mongo.
>
> Also I have another requirement. Whenever data comes to a collection in
> MongoDB, the same data has to move into another collection of a different
> DB in the same server. Can we achieve this using Nifi.
>
> Your help is really appreciated.
>
> Regards
> Kavitha Alahari
>

Re: Need help in getMongo

[Mike Thomsen <mi...@gmail.com>]

1. Have you validated your query? I've been using GetMongo lately for a
fairly large bulk ingestion job, and so far it's worked fine.

2. You can try something like this pattern:
https://stackoverflow.com/questions/44573618/how-to-get-iso-string-in-nifi-getmongo-query-field/44725438#44725438
    * Add a timestamp field to each object.
    * Use a $where with a stored procedure that calculates the last 5
minutes.
    * Set NiFi to run that GetMongo every 5 minutes.

Mike

On Thu, Jun 29, 2017 at 12:39 PM, Kavitha Alahari <Kavitha.Alahari@telus.com
> wrote:

> Hello Team,
>
> I am trying to pull the data from MongoDB and try to put in other Mongo
> DB. For this I used GetMongo and PutMongo processors. Unfortunately none of
> the query is working in getMongo. However it is working when no query is
> there. Could you please provide me a sample query to use in Get Mongo.
>
> Also I have another requirement. Whenever data comes to a collection in
> MongoDB, the same data has to move into another collection of a different
> DB in the same server. Can we achieve this using Nifi.
>
> Your help is really appreciated.
>
> Regards
> Kavitha Alahari
>