You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jungtaek Lim <ka...@gmail.com> on 2022/10/13 02:11:37 UTC
[DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)
Hi all,
I would like to propose flipping the default value of Kafka offset fetching
config. The context is following:
Before Spark 3.1, there was only one approach on fetching offset, using
consumer.poll(0). This has been pointed out as a root cause for hang since
there is no timeout for metadata fetch.
In Spark 3.1, we addressed this via introducing a new approach on fetching
offset, via SPARK-32032 <https://issues.apache.org/jira/browse/SPARK-32032>.
Since the new approach leverages AdminClient and consumer group is no
longer needed for fetching offset, required security ACLs are loosen.
Reference:
https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#offset-fetching
There was some concern about behavioral change on the security model hence
we couldn't make the new approach by default.
During the time, we have observed various Kafka connector related issues
which came from old offset fetching (e.g. hang, issues on rebalance on
customer group, etc.) and we fixed many of these issues via simply flipping
the config.
Based on this, I would consider the default value as "incorrect". The
security-related behavioral change would be introduced inevitably (they can
set topic based ACL rule), but most people will get benefited. IMHO this is
something we can deal with release/migration note.
Would like to hear the voices on this.
Thanks,
Jungtaek Lim (HeartSaVioR)
Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)
Posted by Jungtaek Lim <ka...@gmail.com>.
No further voice so far. I'm going to submit a PR. Thanks again for the
feedback!
On Mon, Oct 17, 2022 at 9:30 AM Jungtaek Lim <ka...@gmail.com>
wrote:
> Thanks Gabor and Dongjoon for supporting this!
>
> Bump to reach more eyes. If there is no further voice on this in a couple
> of days, I'll consider it as a lazy consensus and submit a PR to this.
>
> On Sat, Oct 15, 2022 at 3:32 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> +1
>>
>> I agree with Jungtaek and Gabor about switching the default value of
>> configurations with the migration guide.
>>
>> Dongjoon
>>
>> On Thu, Oct 13, 2022 at 12:46 AM Gabor Somogyi <ga...@gmail.com>
>> wrote:
>>
>>> Hi Jungtaek,
>>>
>>> Good to hear that the new approach is working fine. +1 from my side.
>>>
>>> BR,
>>> G
>>>
>>>
>>> On Thu, Oct 13, 2022 at 4:12 AM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I would like to propose flipping the default value of Kafka offset
>>>> fetching config. The context is following:
>>>>
>>>> Before Spark 3.1, there was only one approach on fetching offset, using
>>>> consumer.poll(0). This has been pointed out as a root cause for hang since
>>>> there is no timeout for metadata fetch.
>>>>
>>>> In Spark 3.1, we addressed this via introducing a new approach on
>>>> fetching offset, via SPARK-32032
>>>> <https://issues.apache.org/jira/browse/SPARK-32032>. Since the new
>>>> approach leverages AdminClient and consumer group is no longer needed for
>>>> fetching offset, required security ACLs are loosen.
>>>>
>>>> Reference:
>>>> https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#offset-fetching
>>>>
>>>> There was some concern about behavioral change on the security model
>>>> hence we couldn't make the new approach by default.
>>>>
>>>> During the time, we have observed various Kafka connector related
>>>> issues which came from old offset fetching (e.g. hang, issues on rebalance
>>>> on customer group, etc.) and we fixed many of these issues via simply
>>>> flipping the config.
>>>>
>>>> Based on this, I would consider the default value as "incorrect". The
>>>> security-related behavioral change would be introduced inevitably (they can
>>>> set topic based ACL rule), but most people will get benefited. IMHO this is
>>>> something we can deal with release/migration note.
>>>>
>>>> Would like to hear the voices on this.
>>>>
>>>> Thanks,
>>>> Jungtaek Lim (HeartSaVioR)
>>>>
>>>
Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks Gabor and Dongjoon for supporting this!
Bump to reach more eyes. If there is no further voice on this in a couple
of days, I'll consider it as a lazy consensus and submit a PR to this.
On Sat, Oct 15, 2022 at 3:32 AM Dongjoon Hyun <do...@gmail.com>
wrote:
> +1
>
> I agree with Jungtaek and Gabor about switching the default value of
> configurations with the migration guide.
>
> Dongjoon
>
> On Thu, Oct 13, 2022 at 12:46 AM Gabor Somogyi <ga...@gmail.com>
> wrote:
>
>> Hi Jungtaek,
>>
>> Good to hear that the new approach is working fine. +1 from my side.
>>
>> BR,
>> G
>>
>>
>> On Thu, Oct 13, 2022 at 4:12 AM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I would like to propose flipping the default value of Kafka offset
>>> fetching config. The context is following:
>>>
>>> Before Spark 3.1, there was only one approach on fetching offset, using
>>> consumer.poll(0). This has been pointed out as a root cause for hang since
>>> there is no timeout for metadata fetch.
>>>
>>> In Spark 3.1, we addressed this via introducing a new approach on
>>> fetching offset, via SPARK-32032
>>> <https://issues.apache.org/jira/browse/SPARK-32032>. Since the new
>>> approach leverages AdminClient and consumer group is no longer needed for
>>> fetching offset, required security ACLs are loosen.
>>>
>>> Reference:
>>> https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#offset-fetching
>>>
>>> There was some concern about behavioral change on the security model
>>> hence we couldn't make the new approach by default.
>>>
>>> During the time, we have observed various Kafka connector related issues
>>> which came from old offset fetching (e.g. hang, issues on rebalance on
>>> customer group, etc.) and we fixed many of these issues via simply flipping
>>> the config.
>>>
>>> Based on this, I would consider the default value as "incorrect". The
>>> security-related behavioral change would be introduced inevitably (they can
>>> set topic based ACL rule), but most people will get benefited. IMHO this is
>>> something we can deal with release/migration note.
>>>
>>> Would like to hear the voices on this.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>
Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)
Posted by Dongjoon Hyun <do...@gmail.com>.
+1
I agree with Jungtaek and Gabor about switching the default value of
configurations with the migration guide.
Dongjoon
On Thu, Oct 13, 2022 at 12:46 AM Gabor Somogyi <ga...@gmail.com>
wrote:
> Hi Jungtaek,
>
> Good to hear that the new approach is working fine. +1 from my side.
>
> BR,
> G
>
>
> On Thu, Oct 13, 2022 at 4:12 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I would like to propose flipping the default value of Kafka offset
>> fetching config. The context is following:
>>
>> Before Spark 3.1, there was only one approach on fetching offset, using
>> consumer.poll(0). This has been pointed out as a root cause for hang since
>> there is no timeout for metadata fetch.
>>
>> In Spark 3.1, we addressed this via introducing a new approach on
>> fetching offset, via SPARK-32032
>> <https://issues.apache.org/jira/browse/SPARK-32032>. Since the new
>> approach leverages AdminClient and consumer group is no longer needed for
>> fetching offset, required security ACLs are loosen.
>>
>> Reference:
>> https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#offset-fetching
>>
>> There was some concern about behavioral change on the security model
>> hence we couldn't make the new approach by default.
>>
>> During the time, we have observed various Kafka connector related issues
>> which came from old offset fetching (e.g. hang, issues on rebalance on
>> customer group, etc.) and we fixed many of these issues via simply flipping
>> the config.
>>
>> Based on this, I would consider the default value as "incorrect". The
>> security-related behavioral change would be introduced inevitably (they can
>> set topic based ACL rule), but most people will get benefited. IMHO this is
>> something we can deal with release/migration note.
>>
>> Would like to hear the voices on this.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>
Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)
Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Jungtaek,
Good to hear that the new approach is working fine. +1 from my side.
BR,
G
On Thu, Oct 13, 2022 at 4:12 AM Jungtaek Lim <ka...@gmail.com>
wrote:
> Hi all,
>
> I would like to propose flipping the default value of Kafka offset
> fetching config. The context is following:
>
> Before Spark 3.1, there was only one approach on fetching offset, using
> consumer.poll(0). This has been pointed out as a root cause for hang since
> there is no timeout for metadata fetch.
>
> In Spark 3.1, we addressed this via introducing a new approach on fetching
> offset, via SPARK-32032
> <https://issues.apache.org/jira/browse/SPARK-32032>. Since the new
> approach leverages AdminClient and consumer group is no longer needed for
> fetching offset, required security ACLs are loosen.
>
> Reference:
> https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#offset-fetching
>
> There was some concern about behavioral change on the security model hence
> we couldn't make the new approach by default.
>
> During the time, we have observed various Kafka connector related issues
> which came from old offset fetching (e.g. hang, issues on rebalance on
> customer group, etc.) and we fixed many of these issues via simply flipping
> the config.
>
> Based on this, I would consider the default value as "incorrect". The
> security-related behavioral change would be introduced inevitably (they can
> set topic based ACL rule), but most people will get benefited. IMHO this is
> something we can deal with release/migration note.
>
> Would like to hear the voices on this.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>