You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by amir bahmanyari <am...@yahoo.com> on 2016/08/12 00:16:40 UTC
Appropriate Kafka & Hadoop versions...
Hi Colleagues,I am rebuilding my Beam lab servers with the latest clusters Flink & eventually Spark.Whats the appropriate versions of Kafka & Hadoop to install so there wont be any potential short ckt?
ThanksAmir-
Re: Appropriate Kafka & Hadoop versions...
Posted by Emanuele Cesena <em...@shopkick.com>.
Hi,
The last part (the jira issue) is about Flink.
(Flink itself should have support for all Kafka versions, but I don’t think that’s the recommended way to go with Beam.)
Best,
> On Aug 11, 2016, at 11:19 PM, Amir Bahmanyari <am...@yahoo.com> wrote:
>
> Thanks Emanuele
> Does this all apply to Flink as well?
> Cheers
>
>
> Sent from my iPhone
>
>> On Aug 11, 2016, at 10:48 PM, Emanuele Cesena <em...@shopkick.com> wrote:
>>
>> Hi Amir,
>>
>> Which version of Spark are you using, 1.6 or 2?
>>
>> On hadoop I don’t think there’s any issue and I’d go with the latest 2.7 (or 2.6 if you have any special requirements).
>>
>> To the best of my knowledge:
>> - Spark 2 works with Kafka 0.9+
>> - Spark 1.6 requires Kafka 0.8.x — although some distro can have different versions, for instance we’re using CDH 5.7 that ships Spark 1.6 working with Kafka 0.9.
>>
>> If you use/plan to use Beam’s KafkaIO then it requires Kafka 0.9+ (I’d personally go with 0.10), but the last time I tried it with Flink it wasn’t fully working because of this:
>> https://issues.apache.org/jira/browse/BEAM-102
>> (Aljoscha told me he was working on it and we can expect a patch soon)
>>
>> This is all I know.
>>
>> Best,
>>
>>
>>> On Aug 11, 2016, at 5:16 PM, amir bahmanyari <am...@yahoo.com> wrote:
>>>
>>> Hi Colleagues,
>>> I am rebuilding my Beam lab servers with the latest clusters Flink & eventually Spark.
>>> Whats the appropriate versions of Kafka & Hadoop to install so there wont be any potential short ckt?
>>>
>>> Thanks
>>> Amir-
>>
>> --
>> Emanuele Cesena, Data Eng.
>> http://www.shopkick.com
>>
>> Il corpo non ha ideali
>>
>>
>>
>>
>
--
Emanuele Cesena, Data Eng.
http://www.shopkick.com
Il corpo non ha ideali
Re: Appropriate Kafka & Hadoop versions...
Posted by Amir Bahmanyari <am...@yahoo.com>.
Thanks Emanuele
Does this all apply to Flink as well?
Cheers
Sent from my iPhone
> On Aug 11, 2016, at 10:48 PM, Emanuele Cesena <em...@shopkick.com> wrote:
>
> Hi Amir,
>
> Which version of Spark are you using, 1.6 or 2?
>
> On hadoop I don’t think there’s any issue and I’d go with the latest 2.7 (or 2.6 if you have any special requirements).
>
> To the best of my knowledge:
> - Spark 2 works with Kafka 0.9+
> - Spark 1.6 requires Kafka 0.8.x — although some distro can have different versions, for instance we’re using CDH 5.7 that ships Spark 1.6 working with Kafka 0.9.
>
> If you use/plan to use Beam’s KafkaIO then it requires Kafka 0.9+ (I’d personally go with 0.10), but the last time I tried it with Flink it wasn’t fully working because of this:
> https://issues.apache.org/jira/browse/BEAM-102
> (Aljoscha told me he was working on it and we can expect a patch soon)
>
> This is all I know.
>
> Best,
>
>
>> On Aug 11, 2016, at 5:16 PM, amir bahmanyari <am...@yahoo.com> wrote:
>>
>> Hi Colleagues,
>> I am rebuilding my Beam lab servers with the latest clusters Flink & eventually Spark.
>> Whats the appropriate versions of Kafka & Hadoop to install so there wont be any potential short ckt?
>>
>> Thanks
>> Amir-
>
> --
> Emanuele Cesena, Data Eng.
> http://www.shopkick.com
>
> Il corpo non ha ideali
>
>
>
>
Re: Appropriate Kafka & Hadoop versions...
Posted by Emanuele Cesena <em...@shopkick.com>.
Hi Amir,
Which version of Spark are you using, 1.6 or 2?
On hadoop I don’t think there’s any issue and I’d go with the latest 2.7 (or 2.6 if you have any special requirements).
To the best of my knowledge:
- Spark 2 works with Kafka 0.9+
- Spark 1.6 requires Kafka 0.8.x — although some distro can have different versions, for instance we’re using CDH 5.7 that ships Spark 1.6 working with Kafka 0.9.
If you use/plan to use Beam’s KafkaIO then it requires Kafka 0.9+ (I’d personally go with 0.10), but the last time I tried it with Flink it wasn’t fully working because of this:
https://issues.apache.org/jira/browse/BEAM-102
(Aljoscha told me he was working on it and we can expect a patch soon)
This is all I know.
Best,
> On Aug 11, 2016, at 5:16 PM, amir bahmanyari <am...@yahoo.com> wrote:
>
> Hi Colleagues,
> I am rebuilding my Beam lab servers with the latest clusters Flink & eventually Spark.
> Whats the appropriate versions of Kafka & Hadoop to install so there wont be any potential short ckt?
>
> Thanks
> Amir-
--
Emanuele Cesena, Data Eng.
http://www.shopkick.com
Il corpo non ha ideali