You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by sudeep mishra <su...@gmail.com> on 2016/02/25 04:43:57 UTC

How to combine data from multiple processors?

Hi,

I am trying to build a data flow where I want outputs of two ExecuteSQL
processors to be combined and add certain extra metadata. The resulting
data should be in JSON format.

What is a good approach to achieve this?

Can I use the funnel component for this purpose?

Thanks & Regards,

Sudeep

Re: How to combine data from multiple processors?

Posted by sudeep mishra <su...@gmail.com>.

Thanks Joe for your feedback.

On Thu, Feb 25, 2016 at 9:20 AM, Joe Witt <jo...@gmail.com> wrote:

> Sudeep
>
> A funnel simply allows you to take multiple connections and combine
> them into a single connection.  Think of it as merging queues into a
> single queue.  This is useful when you want to control prioritization,
> simplify connection logic to some downstream component, etc..
>
> For the case you're describing you would need to write a custom
> process - at this time.  The general pattern for this can be found in
> MergeContent although it is probably a lot more complicated than
> you'll need for this.  You could also consider using MergeContent
> as-is to generate an archive of the two files combined using some
> correlation attribute.  Then you can have your custom process extract
> the data from the archive and do the combination.  That might be a
> good 'right now' answer.
>
> Thanks
> Joe
>
> On Wed, Feb 24, 2016 at 10:43 PM, sudeep mishra
> <su...@gmail.com> wrote:
> > Hi,
> >
> > I am trying to build a data flow where I want outputs of two ExecuteSQL
> > processors to be combined and add certain extra metadata. The resulting
> data
> > should be in JSON format.
> >
> > What is a good approach to achieve this?
> >
> > Can I use the funnel component for this purpose?
> >
> > Thanks & Regards,
> >
> > Sudeep
>



-- 
Thanks & Regards,

Sudeep

Re: How to combine data from multiple processors?

Posted by Joe Witt <jo...@gmail.com>.

Sudeep

A funnel simply allows you to take multiple connections and combine
them into a single connection.  Think of it as merging queues into a
single queue.  This is useful when you want to control prioritization,
simplify connection logic to some downstream component, etc..

For the case you're describing you would need to write a custom
process - at this time.  The general pattern for this can be found in
MergeContent although it is probably a lot more complicated than
you'll need for this.  You could also consider using MergeContent
as-is to generate an archive of the two files combined using some
correlation attribute.  Then you can have your custom process extract
the data from the archive and do the combination.  That might be a
good 'right now' answer.

Thanks
Joe

On Wed, Feb 24, 2016 at 10:43 PM, sudeep mishra
<su...@gmail.com> wrote:
> Hi,
>
> I am trying to build a data flow where I want outputs of two ExecuteSQL
> processors to be combined and add certain extra metadata. The resulting data
> should be in JSON format.
>
> What is a good approach to achieve this?
>
> Can I use the funnel component for this purpose?
>
> Thanks & Regards,
>
> Sudeep