You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Gunjan Dave <gu...@gmail.com> on 2016/09/06 13:03:03 UTC
What next with NiFi
Hello NiFi team, now that version 1.0.0 is out in open, what are the next
big ticket plans?
I saw 1.1.0 jiras but those are mostly bug fixes and minor enhacements, but
what are the larger plans from the existing roadmap?
Re: What next with NiFi
Posted by Tijo Thomas <ti...@gmail.com>.
Hi Joe,
I am not sure whether I sounds stupid. I was also researching on the
option 2.
I felt nifi lack the capability to process the big data jobs where data
need to be processed with locality.
Initially I was thinking about making the modification to run nifi on
yarn. But this approach also will not solve many of the problems like
splitting input for parallel execution etc. Of course we can write
processor to do that. But I wish the frame work should take a call on
that, not processor.
I got lost when I thought about running nifi on other framework like
flink. But I feel beam is a better option .
I also feel that nifi as a beam runner is not a good idea. May be I am not
aware what we will gain out of this.
Earlier I used aws pipeline and Azure data factory, I felt the concept
of data source node is a good and easy to use. Do we have a plan to
introduce the data source concept.
Tijo
On 06-Sep-2016 7:38 pm, "Joe Witt" <jo...@gmail.com> wrote:
> Gunjan
>
> No plans at this point. What sort of beam related integration are you
> envisioning? There are a couple scenarios that come to mind:
> 1) NiFi as a beam runner. Probably not a great fit as by design we'd
> not be attempting to address and important cross section of the types
> of processing you could define in a beam API. Storm/Flink/Spark
> already have those or are under way and we'd just integrate there.
>
> 2) NiFi being able to submit processing flows to Beam. No idea what
> this would really mean yet.
>
> Thanks
> Joe
>
> On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave <gu...@gmail.com>
> wrote:
> > Thank Joe,
> > w.r.t to NiFi externals, like we currently have NiFi spark and storm
> > externals, any plans for NiFi beam external? Reason is I saw your name as
> > one of the contributors there in incubation proposal.
> > I was studying beam in more depth hence the question.
> >
> >
> > On Tue, Sep 6, 2016, 6:42 PM Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Gunjan
> >>
> >> The best indicator of areas of focus I think are listed here
> >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> >>
> >> In a roadmap discussion thread back in January of this year the items
> >> mentioned specifically as trailing the 1.0 release were extension and
> >> variable registries.
> >>
> >> Which items will get addressed in which order is perhaps less precise
> >> and more a function of where contributions occur.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gunjanpiyushdave@gmail.com
> >
> >> wrote:
> >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the
> >> > next
> >> > big ticket plans?
> >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor
> enhacements,
> >> > but
> >> > what are the larger plans from the existing roadmap?
> >> >
>
Re: What next with NiFi
Posted by Bryan Bende <bb...@gmail.com>.
Gunjan/Joe,
I was looking into this at one point... I think the idea would be to
implement a Beam source/sink (whatever their terminology is) for NiFi, and
this would allow someone to write a single Beam job pulling/pushing data
to/from NiFi, and then that single job could be executed in Flink, Storm,
or whatever underlying providers are available to Beam.
For example, here is their Kafka integration:
https://github.com/apache/incubator-beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
-Bryan
On Tue, Sep 6, 2016 at 10:07 AM, Joe Witt <jo...@gmail.com> wrote:
> Gunjan
>
> No plans at this point. What sort of beam related integration are you
> envisioning? There are a couple scenarios that come to mind:
> 1) NiFi as a beam runner. Probably not a great fit as by design we'd
> not be attempting to address and important cross section of the types
> of processing you could define in a beam API. Storm/Flink/Spark
> already have those or are under way and we'd just integrate there.
>
> 2) NiFi being able to submit processing flows to Beam. No idea what
> this would really mean yet.
>
> Thanks
> Joe
>
> On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave <gu...@gmail.com>
> wrote:
> > Thank Joe,
> > w.r.t to NiFi externals, like we currently have NiFi spark and storm
> > externals, any plans for NiFi beam external? Reason is I saw your name as
> > one of the contributors there in incubation proposal.
> > I was studying beam in more depth hence the question.
> >
> >
> > On Tue, Sep 6, 2016, 6:42 PM Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Gunjan
> >>
> >> The best indicator of areas of focus I think are listed here
> >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> >>
> >> In a roadmap discussion thread back in January of this year the items
> >> mentioned specifically as trailing the 1.0 release were extension and
> >> variable registries.
> >>
> >> Which items will get addressed in which order is perhaps less precise
> >> and more a function of where contributions occur.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gunjanpiyushdave@gmail.com
> >
> >> wrote:
> >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the
> >> > next
> >> > big ticket plans?
> >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor
> enhacements,
> >> > but
> >> > what are the larger plans from the existing roadmap?
> >> >
>
Re: What next with NiFi
Posted by Joe Witt <jo...@gmail.com>.
Gunjan
No plans at this point. What sort of beam related integration are you
envisioning? There are a couple scenarios that come to mind:
1) NiFi as a beam runner. Probably not a great fit as by design we'd
not be attempting to address and important cross section of the types
of processing you could define in a beam API. Storm/Flink/Spark
already have those or are under way and we'd just integrate there.
2) NiFi being able to submit processing flows to Beam. No idea what
this would really mean yet.
Thanks
Joe
On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave <gu...@gmail.com> wrote:
> Thank Joe,
> w.r.t to NiFi externals, like we currently have NiFi spark and storm
> externals, any plans for NiFi beam external? Reason is I saw your name as
> one of the contributors there in incubation proposal.
> I was studying beam in more depth hence the question.
>
>
> On Tue, Sep 6, 2016, 6:42 PM Joe Witt <jo...@gmail.com> wrote:
>>
>> Gunjan
>>
>> The best indicator of areas of focus I think are listed here
>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
>>
>> In a roadmap discussion thread back in January of this year the items
>> mentioned specifically as trailing the 1.0 release were extension and
>> variable registries.
>>
>> Which items will get addressed in which order is perhaps less precise
>> and more a function of where contributions occur.
>>
>> Thanks
>> Joe
>>
>> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gu...@gmail.com>
>> wrote:
>> > Hello NiFi team, now that version 1.0.0 is out in open, what are the
>> > next
>> > big ticket plans?
>> > I saw 1.1.0 jiras but those are mostly bug fixes and minor enhacements,
>> > but
>> > what are the larger plans from the existing roadmap?
>> >
Re: What next with NiFi
Posted by Gunjan Dave <gu...@gmail.com>.
Thank Joe,
w.r.t to NiFi externals, like we currently have NiFi spark and storm
externals, any plans for NiFi beam external? Reason is I saw your name as
one of the contributors there in incubation proposal.
I was studying beam in more depth hence the question.
On Tue, Sep 6, 2016, 6:42 PM Joe Witt <jo...@gmail.com> wrote:
> Gunjan
>
> The best indicator of areas of focus I think are listed here
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
>
> In a roadmap discussion thread back in January of this year the items
> mentioned specifically as trailing the 1.0 release were extension and
> variable registries.
>
> Which items will get addressed in which order is perhaps less precise
> and more a function of where contributions occur.
>
> Thanks
> Joe
>
> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gu...@gmail.com>
> wrote:
> > Hello NiFi team, now that version 1.0.0 is out in open, what are the next
> > big ticket plans?
> > I saw 1.1.0 jiras but those are mostly bug fixes and minor enhacements,
> but
> > what are the larger plans from the existing roadmap?
> >
>
Re: What next with NiFi
Posted by Bryan Bende <bb...@gmail.com>.
Ram,
There was some design discussion about this here [1], see the part about
"getAuthorizationHistory", but I don't think we currently have any JIRAs
defined.
The file-based authorizer provided in the 1.0.0 release does not provide
this capability yet, so you would need to implement a custom authorizer
that wrapped the file-based authorizer and then did something with the
results.
Another option is to use Ranger as the authorizer for NiFi, Ranger provides
auditing and typically stores the audit logs in Solr.
-Bryan
[1] https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API
On Tue, Sep 6, 2016 at 10:24 AM, Nathamuni, Ramanujam <RN...@tiaa.org>
wrote:
> Hi Bryan,
>
>
>
> Thanks, Yes. How do we get the audit log from NiFi? I need to integrate
> the audit log with IBM aveksa or IBM Guradium every day.
>
>
>
> Thanks,
>
> Ram
>
>
>
> *From:* Bryan Bende [mailto:bbende@gmail.com]
> *Sent:* Tuesday, September 06, 2016 10:20 AM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: What next with NiFi
>
>
>
> Ram,
>
>
>
> Are you talking about isolation from a security perspective? If so, that
> is now available in the 1.0.0 release.
>
>
>
> You can restrict access to any portion of the flow through a policy model.
>
>
>
> -Bryan
>
>
>
> On Tue, Sep 6, 2016 at 10:16 AM, Nathamuni, Ramanujam <RN...@tiaa.org>
> wrote:
>
> Hi Joe and Team:
>
> Enterprise needs very high data security and audit trial. What is the
> vision to have canvas/processor/processor group ...etc. isolated for
> project/LOBs?
>
> Thanks,
> Ram
>
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Tuesday, September 06, 2016 9:12 AM
> To: users@nifi.apache.org
> Subject: Re: What next with NiFi
>
> Gunjan
>
> The best indicator of areas of focus I think are listed here
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
>
> In a roadmap discussion thread back in January of this year the items
> mentioned specifically as trailing the 1.0 release were extension and
> variable registries.
>
> Which items will get addressed in which order is perhaps less precise and
> more a function of where contributions occur.
>
> Thanks
> Joe
>
> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gu...@gmail.com>
> wrote:
> > Hello NiFi team, now that version 1.0.0 is out in open, what are the
> > next big ticket plans?
> > I saw 1.1.0 jiras but those are mostly bug fixes and minor
> > enhacements, but what are the larger plans from the existing roadmap?
> >
>
> *************************************************************************
> This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender
> immediately and then delete it.
>
> TIAA
> *************************************************************************
>
>
>
>
> *************************************************************************
> This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender
> immediately and then delete it.
>
> TIAA
> *************************************************************************
>
RE: What next with NiFi
Posted by "Nathamuni, Ramanujam" <RN...@tiaa.org>.
Hi Bryan,
Thanks, Yes. How do we get the audit log from NiFi? I need to integrate the audit log with IBM aveksa or IBM Guradium every day.
Thanks,
Ram
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Tuesday, September 06, 2016 10:20 AM
To: users@nifi.apache.org
Subject: Re: What next with NiFi
Ram,
Are you talking about isolation from a security perspective? If so, that is now available in the 1.0.0 release.
You can restrict access to any portion of the flow through a policy model.
-Bryan
On Tue, Sep 6, 2016 at 10:16 AM, Nathamuni, Ramanujam <RN...@tiaa.org>> wrote:
Hi Joe and Team:
Enterprise needs very high data security and audit trial. What is the vision to have canvas/processor/processor group ...etc. isolated for project/LOBs?
Thanks,
Ram
-----Original Message-----
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Tuesday, September 06, 2016 9:12 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: What next with NiFi
Gunjan
The best indicator of areas of focus I think are listed here https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
In a roadmap discussion thread back in January of this year the items mentioned specifically as trailing the 1.0 release were extension and variable registries.
Which items will get addressed in which order is perhaps less precise and more a function of where contributions occur.
Thanks
Joe
On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gu...@gmail.com>> wrote:
> Hello NiFi team, now that version 1.0.0 is out in open, what are the
> next big ticket plans?
> I saw 1.1.0 jiras but those are mostly bug fixes and minor
> enhacements, but what are the larger plans from the existing roadmap?
>
*************************************************************************
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and then delete it.
TIAA
*************************************************************************
*************************************************************************
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and then delete it.
TIAA
*************************************************************************
Re: What next with NiFi
Posted by Bryan Bende <bb...@gmail.com>.
Ram,
Are you talking about isolation from a security perspective? If so, that is
now available in the 1.0.0 release.
You can restrict access to any portion of the flow through a policy model.
-Bryan
On Tue, Sep 6, 2016 at 10:16 AM, Nathamuni, Ramanujam <RN...@tiaa.org>
wrote:
> Hi Joe and Team:
>
> Enterprise needs very high data security and audit trial. What is the
> vision to have canvas/processor/processor group ...etc. isolated for
> project/LOBs?
>
> Thanks,
> Ram
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Tuesday, September 06, 2016 9:12 AM
> To: users@nifi.apache.org
> Subject: Re: What next with NiFi
>
> Gunjan
>
> The best indicator of areas of focus I think are listed here
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
>
> In a roadmap discussion thread back in January of this year the items
> mentioned specifically as trailing the 1.0 release were extension and
> variable registries.
>
> Which items will get addressed in which order is perhaps less precise and
> more a function of where contributions occur.
>
> Thanks
> Joe
>
> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gu...@gmail.com>
> wrote:
> > Hello NiFi team, now that version 1.0.0 is out in open, what are the
> > next big ticket plans?
> > I saw 1.1.0 jiras but those are mostly bug fixes and minor
> > enhacements, but what are the larger plans from the existing roadmap?
> >
>
> *************************************************************************
> This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender
> immediately and then delete it.
>
> TIAA
> *************************************************************************
>
RE: What next with NiFi
Posted by "Nathamuni, Ramanujam" <RN...@tiaa.org>.
Hi Joe and Team:
Enterprise needs very high data security and audit trial. What is the vision to have canvas/processor/processor group ...etc. isolated for project/LOBs?
Thanks,
Ram
-----Original Message-----
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Tuesday, September 06, 2016 9:12 AM
To: users@nifi.apache.org
Subject: Re: What next with NiFi
Gunjan
The best indicator of areas of focus I think are listed here https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
In a roadmap discussion thread back in January of this year the items mentioned specifically as trailing the 1.0 release were extension and variable registries.
Which items will get addressed in which order is perhaps less precise and more a function of where contributions occur.
Thanks
Joe
On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gu...@gmail.com> wrote:
> Hello NiFi team, now that version 1.0.0 is out in open, what are the
> next big ticket plans?
> I saw 1.1.0 jiras but those are mostly bug fixes and minor
> enhacements, but what are the larger plans from the existing roadmap?
>
*************************************************************************
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and then delete it.
TIAA
*************************************************************************
Re: What next with NiFi
Posted by Joe Witt <jo...@gmail.com>.
Gunjan
The best indicator of areas of focus I think are listed here
https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
In a roadmap discussion thread back in January of this year the items
mentioned specifically as trailing the 1.0 release were extension and
variable registries.
Which items will get addressed in which order is perhaps less precise
and more a function of where contributions occur.
Thanks
Joe
On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gu...@gmail.com> wrote:
> Hello NiFi team, now that version 1.0.0 is out in open, what are the next
> big ticket plans?
> I saw 1.1.0 jiras but those are mostly bug fixes and minor enhacements, but
> what are the larger plans from the existing roadmap?
>