You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2019/08/26 11:03:24 UTC
[SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for
initial offsets?
Hi,
Just found out that KafkaSource [1] does not
use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
initial offsets.
Any reason for that? Should I report an issue? Just checking out as I'm
with 2.4.3 exclusively and have no idea what's coming for 3.0.
[1]
https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102
[2]
https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski
Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for
initial offsets?
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
We were worried about regression when adding Kafka source v2 because it had
lots of changes. Hence we copy-pasted codes to keep the Kafka source v1
untouched and provided a config to fallback to v1.
On Mon, Aug 26, 2019 at 7:05 AM Jungtaek Lim <ka...@gmail.com> wrote:
> Thanks! The patch is here: https://github.com/apache/spark/pull/25583
>
> On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
>> Just checked this and it's a copy-paste :) It works properly when
>> KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
>>
>> BR,
>> G
>>
>>
>> On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim <ka...@gmail.com> wrote:
>>
>>> Nice finding! I don't see any reason to not use
>>> KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
>>> guess it was copied and pasted sometime before and not addressed yet.
>>> As you haven't submit a patch, I'll submit a patch shortly, with
>>> mentioning credit. I'd close mine and wait for your patch if you plan to do
>>> it. Please let me know.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>>
>>> On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <ja...@japila.pl> wrote:
>>>
>>>> Hi,
>>>>
>>>> Just found out that KafkaSource [1] does not
>>>> use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
>>>> initial offsets.
>>>>
>>>> Any reason for that? Should I report an issue? Just checking out as I'm
>>>> with 2.4.3 exclusively and have no idea what's coming for 3.0.
>>>>
>>>> [1]
>>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102
>>>>
>>>> [2]
>>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281
>>>>
>>>> Pozdrawiam,
>>>> Jacek Laskowski
>>>> ----
>>>> https://about.me/JacekLaskowski
>>>> The Internals of Spark SQL https://bit.ly/spark-sql-internals
>>>> The Internals of Spark Structured Streaming
>>>> https://bit.ly/spark-structured-streaming
>>>> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>
>>>>
>>>
>>> --
>>> Name : Jungtaek Lim
>>> Blog : http://medium.com/@heartsavior
>>> Twitter : http://twitter.com/heartsavior
>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>
>>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
--
Best Regards,
Ryan
Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for
initial offsets?
Posted by Gabor Somogyi <gs...@cloudera.com.INVALID>.
OK, starting with this tomorrow...
On Mon, 26 Aug 2019, 16:05 Jungtaek Lim, <ka...@gmail.com> wrote:
> Thanks! The patch is here: https://github.com/apache/spark/pull/25583
>
> On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
>> Just checked this and it's a copy-paste :) It works properly when
>> KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
>>
>> BR,
>> G
>>
>>
>> On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim <ka...@gmail.com> wrote:
>>
>>> Nice finding! I don't see any reason to not use
>>> KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
>>> guess it was copied and pasted sometime before and not addressed yet.
>>> As you haven't submit a patch, I'll submit a patch shortly, with
>>> mentioning credit. I'd close mine and wait for your patch if you plan to do
>>> it. Please let me know.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>>
>>> On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <ja...@japila.pl> wrote:
>>>
>>>> Hi,
>>>>
>>>> Just found out that KafkaSource [1] does not
>>>> use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
>>>> initial offsets.
>>>>
>>>> Any reason for that? Should I report an issue? Just checking out as I'm
>>>> with 2.4.3 exclusively and have no idea what's coming for 3.0.
>>>>
>>>> [1]
>>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102
>>>>
>>>> [2]
>>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281
>>>>
>>>> Pozdrawiam,
>>>> Jacek Laskowski
>>>> ----
>>>> https://about.me/JacekLaskowski
>>>> The Internals of Spark SQL https://bit.ly/spark-sql-internals
>>>> The Internals of Spark Structured Streaming
>>>> https://bit.ly/spark-structured-streaming
>>>> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>
>>>>
>>>
>>> --
>>> Name : Jungtaek Lim
>>> Blog : http://medium.com/@heartsavior
>>> Twitter : http://twitter.com/heartsavior
>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>
>>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for
initial offsets?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Devs,
Thanks all for a very prompt response! That was insanely quick. Merci
beaucoup! :)
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski
On Mon, Aug 26, 2019 at 4:05 PM Jungtaek Lim <ka...@gmail.com> wrote:
> Thanks! The patch is here: https://github.com/apache/spark/pull/25583
>
> On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
>> Just checked this and it's a copy-paste :) It works properly when
>> KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
>>
>> BR,
>> G
>>
>>
>> On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim <ka...@gmail.com> wrote:
>>
>>> Nice finding! I don't see any reason to not use
>>> KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
>>> guess it was copied and pasted sometime before and not addressed yet.
>>> As you haven't submit a patch, I'll submit a patch shortly, with
>>> mentioning credit. I'd close mine and wait for your patch if you plan to do
>>> it. Please let me know.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>>
>>> On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <ja...@japila.pl> wrote:
>>>
>>>> Hi,
>>>>
>>>> Just found out that KafkaSource [1] does not
>>>> use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
>>>> initial offsets.
>>>>
>>>> Any reason for that? Should I report an issue? Just checking out as I'm
>>>> with 2.4.3 exclusively and have no idea what's coming for 3.0.
>>>>
>>>> [1]
>>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102
>>>>
>>>> [2]
>>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281
>>>>
>>>> Pozdrawiam,
>>>> Jacek Laskowski
>>>> ----
>>>> https://about.me/JacekLaskowski
>>>> The Internals of Spark SQL https://bit.ly/spark-sql-internals
>>>> The Internals of Spark Structured Streaming
>>>> https://bit.ly/spark-structured-streaming
>>>> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>
>>>>
>>>
>>> --
>>> Name : Jungtaek Lim
>>> Blog : http://medium.com/@heartsavior
>>> Twitter : http://twitter.com/heartsavior
>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>
>>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for
initial offsets?
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks! The patch is here: https://github.com/apache/spark/pull/25583
On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi <ga...@gmail.com>
wrote:
> Just checked this and it's a copy-paste :) It works properly when
> KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
>
> BR,
> G
>
>
> On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim <ka...@gmail.com> wrote:
>
>> Nice finding! I don't see any reason to not use
>> KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
>> guess it was copied and pasted sometime before and not addressed yet.
>> As you haven't submit a patch, I'll submit a patch shortly, with
>> mentioning credit. I'd close mine and wait for your patch if you plan to do
>> it. Please let me know.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>> On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <ja...@japila.pl> wrote:
>>
>>> Hi,
>>>
>>> Just found out that KafkaSource [1] does not
>>> use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
>>> initial offsets.
>>>
>>> Any reason for that? Should I report an issue? Just checking out as I'm
>>> with 2.4.3 exclusively and have no idea what's coming for 3.0.
>>>
>>> [1]
>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102
>>>
>>> [2]
>>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://about.me/JacekLaskowski
>>> The Internals of Spark SQL https://bit.ly/spark-sql-internals
>>> The Internals of Spark Structured Streaming
>>> https://bit.ly/spark-structured-streaming
>>> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>>
>>
>> --
>> Name : Jungtaek Lim
>> Blog : http://medium.com/@heartsavior
>> Twitter : http://twitter.com/heartsavior
>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>
>
--
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for
initial offsets?
Posted by Gabor Somogyi <ga...@gmail.com>.
Just checked this and it's a copy-paste :) It works properly when
KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
BR,
G
On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim <ka...@gmail.com> wrote:
> Nice finding! I don't see any reason to not use
> KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
> guess it was copied and pasted sometime before and not addressed yet.
> As you haven't submit a patch, I'll submit a patch shortly, with
> mentioning credit. I'd close mine and wait for your patch if you plan to do
> it. Please let me know.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi,
>>
>> Just found out that KafkaSource [1] does not
>> use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
>> initial offsets.
>>
>> Any reason for that? Should I report an issue? Just checking out as I'm
>> with 2.4.3 exclusively and have no idea what's coming for 3.0.
>>
>> [1]
>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102
>>
>> [2]
>> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://about.me/JacekLaskowski
>> The Internals of Spark SQL https://bit.ly/spark-sql-internals
>> The Internals of Spark Structured Streaming
>> https://bit.ly/spark-structured-streaming
>> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for
initial offsets?
Posted by Jungtaek Lim <ka...@gmail.com>.
Nice finding! I don't see any reason to not use
KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
guess it was copied and pasted sometime before and not addressed yet.
As you haven't submit a patch, I'll submit a patch shortly, with mentioning
credit. I'd close mine and wait for your patch if you plan to do it. Please
let me know.
Thanks,
Jungtaek Lim (HeartSaVioR)
On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> Just found out that KafkaSource [1] does not
> use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
> initial offsets.
>
> Any reason for that? Should I report an issue? Just checking out as I'm
> with 2.4.3 exclusively and have no idea what's coming for 3.0.
>
> [1]
> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102
>
> [2]
> https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> The Internals of Spark SQL https://bit.ly/spark-sql-internals
> The Internals of Spark Structured Streaming
> https://bit.ly/spark-structured-streaming
> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
> Follow me at https://twitter.com/jaceklaskowski
>
>
--
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior