You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ambari.apache.org by Benjamin Delespierre <bd...@adserverpub.com> on 2015/07/21 17:13:31 UTC

HDP stack 2.3 and Flume 1.6

Hello,

We're using Kafka for event retention and would like to stream the data
directly into Hive using Flume. The problem is that only Flume 1.6 can acts
as a Kafka client for now and it's not yet in the Hortonworks stack
(currently HDP 2.3 still use the 1.5).

How can I get the proper flume version installed ?

- Should I create a custom repository with my package in it (can't get my
hands on a .rpm or .deb for this version by the way) ?
- Should I create a custom service ?
- Should I modify the existing service in order to change the installation
behavior ?
- Am I completely wrong ?

Any help is welcome ;-)

Best regards.
-- Benjamin

Re: HDP stack 2.3 and Flume 1.6

Posted by Benjamin Delespierre <bd...@adserverpub.com>.

We managed to workaround the issue by rebuidling a package and using a
custom Hortonworks repository with Ambari. Everything's just fine :-)

*Benjamin Delespierre **| Developper*
bdelespierre@adserverpub.com
Skype : benjamin.delespierre.75
Hangout : bdelespierre@adserverpub.com
21 rue de Châtillon - 75014 Paris - France
Facebook <http://www.facebook.com/AdServerPub> | Twitter
<http://twitter.com/AdServerPub> | AdServerPub.com
<http://www.adserverpub.com/>

2015-07-28 4:48 GMT+02:00 Artem Ervits <ar...@gmail.com>:

> I faced with similar issue. I was using Storm and Hive. Take a look at
> Hive streaming injest API, it uses the same technique as flume Hive sink.
> On Jul 21, 2015 11:13 AM, "Benjamin Delespierre" <
> bdelespierre@adserverpub.com> wrote:
>
>> Hello,
>>
>> We're using Kafka for event retention and would like to stream the data
>> directly into Hive using Flume. The problem is that only Flume 1.6 can acts
>> as a Kafka client for now and it's not yet in the Hortonworks stack
>> (currently HDP 2.3 still use the 1.5).
>>
>> How can I get the proper flume version installed ?
>>
>> - Should I create a custom repository with my package in it (can't get my
>> hands on a .rpm or .deb for this version by the way) ?
>> - Should I create a custom service ?
>> - Should I modify the existing service in order to change the
>> installation behavior ?
>> - Am I completely wrong ?
>>
>> Any help is welcome ;-)
>>
>> Best regards.
>> -- Benjamin
>>
>>

Re: HDP stack 2.3 and Flume 1.6

Posted by Artem Ervits <ar...@gmail.com>.

I faced with similar issue. I was using Storm and Hive. Take a look at Hive
streaming injest API, it uses the same technique as flume Hive sink.
On Jul 21, 2015 11:13 AM, "Benjamin Delespierre" <
bdelespierre@adserverpub.com> wrote:

> Hello,
>
> We're using Kafka for event retention and would like to stream the data
> directly into Hive using Flume. The problem is that only Flume 1.6 can acts
> as a Kafka client for now and it's not yet in the Hortonworks stack
> (currently HDP 2.3 still use the 1.5).
>
> How can I get the proper flume version installed ?
>
> - Should I create a custom repository with my package in it (can't get my
> hands on a .rpm or .deb for this version by the way) ?
> - Should I create a custom service ?
> - Should I modify the existing service in order to change the installation
> behavior ?
> - Am I completely wrong ?
>
> Any help is welcome ;-)
>
> Best regards.
> -- Benjamin
>
>