You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Michael Armbrust <mi...@databricks.com> on 2016/10/07 19:41:19 UTC
Kafaka 0.8, 0.9 in Structured Streaming
We recently merged support for Kafak 0.10.0 in Structured Streaming, but
I've been hearing a few people tell me that they are stuck on an older
version of Kafka and cannot upgrade. I'm considering revisiting SPARK-17344
<https://issues.apache.org/jira/browse/SPARK-17344>, but it would be good
to have more information.
Could people please vote or comment on the above ticket if a lack of
support for older versions of kafka would block you from trying out
structured streaming?
Thanks!
Michael
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Scott Reynolds <sr...@twilio.com>.
It is always the case that 0.8 and 0.9 will work with a 0.10 broker.
On Fri, Oct 7, 2016 at 1:28 PM Michael Armbrust <mi...@databricks.com>
wrote:
>
> 0.10 consumers won't work on an earlier broker.
> Earlier consumers will (should?) work on a 0.10 broker.
>
>
> This lines up with my testing. Is there a page I'm missing that
> describes this? Like does a 0.9 client work with 0.8 broker? Is it always
> old clients can talk to new brokers but not vice versa?
>
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Michael Armbrust <mi...@databricks.com>.
> 0.10 consumers won't work on an earlier broker.
> Earlier consumers will (should?) work on a 0.10 broker.
>
This lines up with my testing. Is there a page I'm missing that describes
this? Like does a 0.9 client work with 0.8 broker? Is it always old
clients can talk to new brokers but not vice versa?
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Cody Koeninger <co...@koeninger.org>.
The main thing is picking up new partitions. You can't do that
without reimplementing portions of the consumer rebalance. The
low-level consumer is really low level, and the old high-level
consumer is basically broken (it might have been fixed by the time
they abandoned it, I dunno)
On Fri, Oct 7, 2016 at 3:56 PM, Michael Armbrust <mi...@databricks.com> wrote:
>> Without a hell of a lot more work, Assign would be the only strategy
>> usable.
>
>
> How would the current "subscribe" break?
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Michael Armbrust <mi...@databricks.com>.
>
> Without a hell of a lot more work, Assign would be the only strategy
> usable.
How would the current "subscribe" break?
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Cody Koeninger <co...@koeninger.org>.
Without a hell of a lot more work, Assign would be the only strategy usable.
On Fri, Oct 7, 2016 at 3:25 PM, Michael Armbrust <mi...@databricks.com> wrote:
>> The implementation is totally and completely different however, in ways
>> that leak to the end user.
>
>
> Can you elaborate? Especially in the context of the interface provided by
> structured streaming.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Cody Koeninger <co...@koeninger.org>.
Without a hell of a lot more work, Assign would be the only strategy usable.
On Fri, Oct 7, 2016 at 3:25 PM, Michael Armbrust <mi...@databricks.com> wrote:
>> The implementation is totally and completely different however, in ways
>> that leak to the end user.
>
>
> Can you elaborate? Especially in the context of the interface provided by
> structured streaming.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Michael Armbrust <mi...@databricks.com>.
>
> The implementation is totally and completely different however, in ways
> that leak to the end user.
Can you elaborate? Especially in the context of the interface provided by
structured streaming.
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Michael Armbrust <mi...@databricks.com>.
>
> The implementation is totally and completely different however, in ways
> that leak to the end user.
Can you elaborate? Especially in the context of the interface provided by
structured streaming.
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Michael Armbrust <mi...@databricks.com>.
> 0.10 consumers won't work on an earlier broker.
> Earlier consumers will (should?) work on a 0.10 broker.
>
This lines up with my testing. Is there a page I'm missing that describes
this? Like does a 0.9 client work with 0.8 broker? Is it always old
clients can talk to new brokers but not vice versa?
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Cody Koeninger <co...@koeninger.org>.
0.10 consumers won't work on an earlier broker.
Earlier consumers will (should?) work on a 0.10 broker.
The main things earlier consumers lack from a user perspective is
support for SSL, and pre-fetching messages. The implementation is
totally and completely different however, in ways that leak to the end
user.
On Fri, Oct 7, 2016 at 3:15 PM, Reynold Xin <rx...@databricks.com> wrote:
> Does Kafka 0.10 work on a Kafka 0.8/0.9 cluster?
>
>
> On Fri, Oct 7, 2016 at 1:14 PM, Jeremy Smith <je...@acorns.com>
> wrote:
>>
>> +1
>>
>> We're on CDH, and it will probably be a while before they support Kafka
>> 0.10. At the same time, we don't use their Spark and we're looking forward
>> to upgrading to 2.0.x and using structured streaming.
>>
>> I was just going to write our own Kafka Source implementation which uses
>> the existing KafkaRDD but it would be much easier to get buy-in for an
>> official Spark module.
>>
>> Jeremy
>>
>> On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust <mi...@databricks.com>
>> wrote:
>>>
>>> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
>>> I've been hearing a few people tell me that they are stuck on an older
>>> version of Kafka and cannot upgrade. I'm considering revisiting
>>> SPARK-17344, but it would be good to have more information.
>>>
>>> Could people please vote or comment on the above ticket if a lack of
>>> support for older versions of kafka would block you from trying out
>>> structured streaming?
>>>
>>> Thanks!
>>>
>>> Michael
>>
>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Cody Koeninger <co...@koeninger.org>.
0.10 consumers won't work on an earlier broker.
Earlier consumers will (should?) work on a 0.10 broker.
The main things earlier consumers lack from a user perspective is
support for SSL, and pre-fetching messages. The implementation is
totally and completely different however, in ways that leak to the end
user.
On Fri, Oct 7, 2016 at 3:15 PM, Reynold Xin <rx...@databricks.com> wrote:
> Does Kafka 0.10 work on a Kafka 0.8/0.9 cluster?
>
>
> On Fri, Oct 7, 2016 at 1:14 PM, Jeremy Smith <je...@acorns.com>
> wrote:
>>
>> +1
>>
>> We're on CDH, and it will probably be a while before they support Kafka
>> 0.10. At the same time, we don't use their Spark and we're looking forward
>> to upgrading to 2.0.x and using structured streaming.
>>
>> I was just going to write our own Kafka Source implementation which uses
>> the existing KafkaRDD but it would be much easier to get buy-in for an
>> official Spark module.
>>
>> Jeremy
>>
>> On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust <mi...@databricks.com>
>> wrote:
>>>
>>> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
>>> I've been hearing a few people tell me that they are stuck on an older
>>> version of Kafka and cannot upgrade. I'm considering revisiting
>>> SPARK-17344, but it would be good to have more information.
>>>
>>> Could people please vote or comment on the above ticket if a lack of
>>> support for older versions of kafka would block you from trying out
>>> structured streaming?
>>>
>>> Thanks!
>>>
>>> Michael
>>
>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Reynold Xin <rx...@databricks.com>.
Does Kafka 0.10 work on a Kafka 0.8/0.9 cluster?
On Fri, Oct 7, 2016 at 1:14 PM, Jeremy Smith <je...@acorns.com>
wrote:
> +1
>
> We're on CDH, and it will probably be a while before they support Kafka
> 0.10. At the same time, we don't use their Spark and we're looking forward
> to upgrading to 2.0.x and using structured streaming.
>
> I was just going to write our own Kafka Source implementation which uses
> the existing KafkaRDD but it would be much easier to get buy-in for an
> official Spark module.
>
> Jeremy
>
> On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
>> I've been hearing a few people tell me that they are stuck on an older
>> version of Kafka and cannot upgrade. I'm considering revisiting
>> SPARK-17344 <https://issues.apache.org/jira/browse/SPARK-17344>, but it
>> would be good to have more information.
>>
>> Could people please vote or comment on the above ticket if a lack of
>> support for older versions of kafka would block you from trying out
>> structured streaming?
>>
>> Thanks!
>>
>> Michael
>>
>
>
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Reynold Xin <rx...@databricks.com>.
Does Kafka 0.10 work on a Kafka 0.8/0.9 cluster?
On Fri, Oct 7, 2016 at 1:14 PM, Jeremy Smith <je...@acorns.com>
wrote:
> +1
>
> We're on CDH, and it will probably be a while before they support Kafka
> 0.10. At the same time, we don't use their Spark and we're looking forward
> to upgrading to 2.0.x and using structured streaming.
>
> I was just going to write our own Kafka Source implementation which uses
> the existing KafkaRDD but it would be much easier to get buy-in for an
> official Spark module.
>
> Jeremy
>
> On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
>> I've been hearing a few people tell me that they are stuck on an older
>> version of Kafka and cannot upgrade. I'm considering revisiting
>> SPARK-17344 <https://issues.apache.org/jira/browse/SPARK-17344>, but it
>> would be good to have more information.
>>
>> Could people please vote or comment on the above ticket if a lack of
>> support for older versions of kafka would block you from trying out
>> structured streaming?
>>
>> Thanks!
>>
>> Michael
>>
>
>
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Jeremy Smith <je...@acorns.com>.
+1
We're on CDH, and it will probably be a while before they support Kafka
0.10. At the same time, we don't use their Spark and we're looking forward
to upgrading to 2.0.x and using structured streaming.
I was just going to write our own Kafka Source implementation which uses
the existing KafkaRDD but it would be much easier to get buy-in for an
official Spark module.
Jeremy
On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
> I've been hearing a few people tell me that they are stuck on an older
> version of Kafka and cannot upgrade. I'm considering revisiting
> SPARK-17344 <https://issues.apache.org/jira/browse/SPARK-17344>, but it
> would be good to have more information.
>
> Could people please vote or comment on the above ticket if a lack of
> support for older versions of kafka would block you from trying out
> structured streaming?
>
> Thanks!
>
> Michael
>
Re: Kafaka 0.8, 0.9 in Structured Streaming
Posted by Jeremy Smith <je...@acorns.com>.
+1
We're on CDH, and it will probably be a while before they support Kafka
0.10. At the same time, we don't use their Spark and we're looking forward
to upgrading to 2.0.x and using structured streaming.
I was just going to write our own Kafka Source implementation which uses
the existing KafkaRDD but it would be much easier to get buy-in for an
official Spark module.
Jeremy
On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
> I've been hearing a few people tell me that they are stuck on an older
> version of Kafka and cannot upgrade. I'm considering revisiting
> SPARK-17344 <https://issues.apache.org/jira/browse/SPARK-17344>, but it
> would be good to have more information.
>
> Could people please vote or comment on the above ticket if a lack of
> support for older versions of kafka would block you from trying out
> structured streaming?
>
> Thanks!
>
> Michael
>