You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Andre F de Miranda <af...@fucs.org> on 2015/10/12 15:29:11 UTC
Custom classpath (using MapR-FS in lieu of HDFS)
Hi there,
First post in here so please... you know the drill.
I was wondering: What is the recommended approach to add custom
classes to NiFi on load time?
Reason I ask is simple: Our HDFS store is in fact a MapR-FS store and
as such I need to use the proprietary MapR client and Hadoop JARs on
the client software.
Usually this is achieved via add-on shims, rebuild (using maven
artifacts) and in some rare cases, fully supported out of the box! :-)
I am finding a bit difficult to setup the NiFI <-> MapR linkage
without tainting the NiFi distribution folder structure and was
wondering what is the recommended approach?
I thank you in advance
Re: Custom classpath (using MapR-FS in lieu of HDFS)
Posted by Joe Witt <jo...@gmail.com>.
Epic is your google profile picture.
No problem - we're here to help.
For the Hadoop distros we just need to make it easy for folks to
interact with their distro of choice. You should be able to drag the
HDFS processors onto the graph and select 'distro/provider','version',
etc.. and go to town. Same system should work with multiple
providers.
Thanks
Joe
On Mon, Oct 12, 2015 at 10:16 AM, Andre F de Miranda <af...@fucs.org> wrote:
> Joe,
>
> Epic!
>
> Exactly what I was looking for. (Also, apologies for getting you to
> LMGTFY... :-) )
>
> Cheers
>
> On Tue, Oct 13, 2015 at 1:13 AM, Joe Witt <jo...@gmail.com> wrote:
>> Hello Andre,
>>
>> Found on Github here is someone who already got NiFi working with MapR
>> and documented their approach.
>>
>> https://github.com/bbukacek/nifi-hadoop-libraries-bundle
>>
>> Thanks
>> Joe
>>
>> On Mon, Oct 12, 2015 at 9:46 AM, Oleg Zhurakousky
>> <oz...@hortonworks.com> wrote:
>>> May be I am missing certain details, but are you asking about custom bundle (NAR) or using existing NiFi bundles which would somehow have to delegate to some custom JARs?
>>> I am thinking that if you are using custom client, you may need a custom bundle.
>>>
>>> Oleg
>>>
>>>> On Oct 12, 2015, at 9:29 AM, Andre F de Miranda <af...@fucs.org> wrote:
>>>>
>>>> Hi there,
>>>>
>>>> First post in here so please... you know the drill.
>>>>
>>>> I was wondering: What is the recommended approach to add custom
>>>> classes to NiFi on load time?
>>>>
>>>> Reason I ask is simple: Our HDFS store is in fact a MapR-FS store and
>>>> as such I need to use the proprietary MapR client and Hadoop JARs on
>>>> the client software.
>>>>
>>>> Usually this is achieved via add-on shims, rebuild (using maven
>>>> artifacts) and in some rare cases, fully supported out of the box! :-)
>>>>
>>>> I am finding a bit difficult to setup the NiFI <-> MapR linkage
>>>> without tainting the NiFi distribution folder structure and was
>>>> wondering what is the recommended approach?
>>>>
>>>> I thank you in advance
>>>>
>>>
Re: Custom classpath (using MapR-FS in lieu of HDFS)
Posted by Andre F de Miranda <af...@fucs.org>.
Joe,
Epic!
Exactly what I was looking for. (Also, apologies for getting you to
LMGTFY... :-) )
Cheers
On Tue, Oct 13, 2015 at 1:13 AM, Joe Witt <jo...@gmail.com> wrote:
> Hello Andre,
>
> Found on Github here is someone who already got NiFi working with MapR
> and documented their approach.
>
> https://github.com/bbukacek/nifi-hadoop-libraries-bundle
>
> Thanks
> Joe
>
> On Mon, Oct 12, 2015 at 9:46 AM, Oleg Zhurakousky
> <oz...@hortonworks.com> wrote:
>> May be I am missing certain details, but are you asking about custom bundle (NAR) or using existing NiFi bundles which would somehow have to delegate to some custom JARs?
>> I am thinking that if you are using custom client, you may need a custom bundle.
>>
>> Oleg
>>
>>> On Oct 12, 2015, at 9:29 AM, Andre F de Miranda <af...@fucs.org> wrote:
>>>
>>> Hi there,
>>>
>>> First post in here so please... you know the drill.
>>>
>>> I was wondering: What is the recommended approach to add custom
>>> classes to NiFi on load time?
>>>
>>> Reason I ask is simple: Our HDFS store is in fact a MapR-FS store and
>>> as such I need to use the proprietary MapR client and Hadoop JARs on
>>> the client software.
>>>
>>> Usually this is achieved via add-on shims, rebuild (using maven
>>> artifacts) and in some rare cases, fully supported out of the box! :-)
>>>
>>> I am finding a bit difficult to setup the NiFI <-> MapR linkage
>>> without tainting the NiFi distribution folder structure and was
>>> wondering what is the recommended approach?
>>>
>>> I thank you in advance
>>>
>>
Re: Custom classpath (using MapR-FS in lieu of HDFS)
Posted by Joe Witt <jo...@gmail.com>.
Hello Andre,
Found on Github here is someone who already got NiFi working with MapR
and documented their approach.
https://github.com/bbukacek/nifi-hadoop-libraries-bundle
Thanks
Joe
On Mon, Oct 12, 2015 at 9:46 AM, Oleg Zhurakousky
<oz...@hortonworks.com> wrote:
> May be I am missing certain details, but are you asking about custom bundle (NAR) or using existing NiFi bundles which would somehow have to delegate to some custom JARs?
> I am thinking that if you are using custom client, you may need a custom bundle.
>
> Oleg
>
>> On Oct 12, 2015, at 9:29 AM, Andre F de Miranda <af...@fucs.org> wrote:
>>
>> Hi there,
>>
>> First post in here so please... you know the drill.
>>
>> I was wondering: What is the recommended approach to add custom
>> classes to NiFi on load time?
>>
>> Reason I ask is simple: Our HDFS store is in fact a MapR-FS store and
>> as such I need to use the proprietary MapR client and Hadoop JARs on
>> the client software.
>>
>> Usually this is achieved via add-on shims, rebuild (using maven
>> artifacts) and in some rare cases, fully supported out of the box! :-)
>>
>> I am finding a bit difficult to setup the NiFI <-> MapR linkage
>> without tainting the NiFi distribution folder structure and was
>> wondering what is the recommended approach?
>>
>> I thank you in advance
>>
>
Re: Custom classpath (using MapR-FS in lieu of HDFS)
Posted by Andre F de Miranda <af...@fucs.org>.
Oleg,
Yes, you are on the right track. The question is how to use custom
classes on existing NiFi bundles / release packages without creating a
code mess.
>From what I understand I could simply use Maven to compile a MapR
friendly version of NiFi but some solutions are happy with basic mods
to existing code:
https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/destination-hdfs-maprfs.html
https://www-304.ibm.com/support/knowledgecenter/SSGSMK_7.1.0/mapreduce_integration/mapr.dita
http://answers.mapr.com/questions/163588/reading-maprfs-from-spark.html
Is my question a bit clearer now?
Cheers
On Tue, Oct 13, 2015 at 12:46 AM, Oleg Zhurakousky
<oz...@hortonworks.com> wrote:
> May be I am missing certain details, but are you asking about custom bundle (NAR) or using existing NiFi bundles which would somehow have to delegate to some custom JARs?
> I am thinking that if you are using custom client, you may need a custom bundle.
>
> Oleg
>
>> On Oct 12, 2015, at 9:29 AM, Andre F de Miranda <af...@fucs.org> wrote:
>>
>> Hi there,
>>
>> First post in here so please... you know the drill.
>>
>> I was wondering: What is the recommended approach to add custom
>> classes to NiFi on load time?
>>
>> Reason I ask is simple: Our HDFS store is in fact a MapR-FS store and
>> as such I need to use the proprietary MapR client and Hadoop JARs on
>> the client software.
>>
>> Usually this is achieved via add-on shims, rebuild (using maven
>> artifacts) and in some rare cases, fully supported out of the box! :-)
>>
>> I am finding a bit difficult to setup the NiFI <-> MapR linkage
>> without tainting the NiFi distribution folder structure and was
>> wondering what is the recommended approach?
>>
>> I thank you in advance
>>
>
Re: Custom classpath (using MapR-FS in lieu of HDFS)
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
May be I am missing certain details, but are you asking about custom bundle (NAR) or using existing NiFi bundles which would somehow have to delegate to some custom JARs?
I am thinking that if you are using custom client, you may need a custom bundle.
Oleg
> On Oct 12, 2015, at 9:29 AM, Andre F de Miranda <af...@fucs.org> wrote:
>
> Hi there,
>
> First post in here so please... you know the drill.
>
> I was wondering: What is the recommended approach to add custom
> classes to NiFi on load time?
>
> Reason I ask is simple: Our HDFS store is in fact a MapR-FS store and
> as such I need to use the proprietary MapR client and Hadoop JARs on
> the client software.
>
> Usually this is achieved via add-on shims, rebuild (using maven
> artifacts) and in some rare cases, fully supported out of the box! :-)
>
> I am finding a bit difficult to setup the NiFI <-> MapR linkage
> without tainting the NiFi distribution folder structure and was
> wondering what is the recommended approach?
>
> I thank you in advance
>