[DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Hi dev,

This is a spin-up of the original thread "Deprecating and banning
`spark.databricks.*` config from Apache Spark repository". (link
<https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)

From the original thread, we decided to deprecate the config in Spark 3.5.5
and remove the config in Spark 4.0.0. That thread did not decide one thing,
about smooth migration logic.

We "persist" the config into offset log for streaming query since the value
of the config must be consistent during the lifecycle of the query. This
means, the problematic config is already persisted for streaming query
which ever ran with Spark 3.5.4.

For the migration logic, we re-assign the value of the problematic config
to the new config. This happens when the query is restarted, and it will be
reflected into an offset log for "newer batch" so after a couple new
microbatches the migration logic isn't needed. This migration logic is
shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
couple microbatches, it will be mitigated.

But I would say that there will always be a case that users just bump the
minor/major version without following all the bugfix versions. I think it
is still dangerous to remove the migration logic in Spark 4.0.0 (and
probably Spark 4.1.0, depending on the discussion). From the migration
logic, the problematic config is just a "string", and users wouldn't be
able to set the value with the problematic config name. We don't document
this, as it'll be done automatically.

That said, I'd propose to have migration logic for Spark 4.0 version line
(at minimum, 4.1 is debatable). This will give a safer and less burden
migration path for users with just retaining a problematic "string" (again,
not a config).

I'd love to hear the community's voice on this. I'd like to remind you,
this is a blocker for Spark 4.0.0.

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Nicholas Chammas <ni...@gmail.com>]

> On Mar 10, 2025, at 10:14 PM, Andrew Melo <an...@gmail.com> wrote:
>> 
>> This config was released to "Apache" Spark 3.5.4, so this is NO LONGER just a problem with vendor distribution. The breakage will happen even if someone does not even know about Databricks Runtime at all and keeps using Apache Spark. It just happens when users in Apache Spark 3.5.4 directly upgrade to Apache Spark 4.0.0+ (rather than upgrading once to Apache Spark 3.5.5+).
>> (Again, from the vendor codebase, it is not a problem at all to have such a config and there is a much easier way to deal with it - migration logic is not needed at all.)
> 
> This is code that will be carried for sometime, and regardless of what
> you responded to, I asked about what *users* would be affected. I
> think it's fine to let your response speak for itself.

Andrew, your question about what users are affected is answered directly by Jungtaek in the part you quoted just above.

To restate: It affects open source users of Apache Spark who try to upgrade directly from 3.5.4 to 4.0.0 without first upgrading to 3.5.5.

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Andrew Melo <an...@gmail.com>]

Hi Jungtaek

below

On Mon, Mar 10, 2025 at 9:02 PM Jungtaek Lim
<ka...@gmail.com> wrote:
>
> Replied inline
>
> On Tue, Mar 11, 2025 at 10:39 AM Andrew Melo <an...@gmail.com> wrote:
>>
>> Hi Jungtaek,
>>
>> I've read the discussion, which is why I replied with my questions
>> (which you neglected to answer). Your deflection and lack of response
>> to direct questions should be (IMO) disqualifying. So, again:
>>
>> To put it into less complicated words - presumably the people using
>> the databricks.* configs were already using the databricks runtime.
>> Why does Apache spark need to carry an extra migration patch when the
>> users who would be affected are already using the Databricks fork? I
>> don't see a situation where:
>>
>> A) legacy 3.5.x queries were using databricks-specific options
>> B) these users want to run the same queries in OSS spark today
>> C) the same people will not be using the databricks fork.
>
>
> I don't understand why you couple this with Databricks Runtime.
>
> If I were pushing this to the case where the config was only released in Databricks Runtime and not yet to be released in Apache Spark, you are definitely right, although there are a bunch of other ways around to solve the issue on the vendor side, and I wouldn't choose this way which would be expected to have a lot of burden.
>
> This config was released to "Apache" Spark 3.5.4, so this is NO LONGER just a problem with vendor distribution. The breakage will happen even if someone does not even know about Databricks Runtime at all and keeps using Apache Spark. It just happens when users in Apache Spark 3.5.4 directly upgrade to Apache Spark 4.0.0+ (rather than upgrading once to Apache Spark 3.5.5+).
> (Again, from the vendor codebase, it is not a problem at all to have such a config and there is a much easier way to deal with it - migration logic is not needed at all.)
>
> I think this is a major misunderstanding that you see in this discussion to push vendor's code into OSS. Again, the migration logic is needed only by OSS. I can guarantee that I am not doing this for a vendor. (I made my first ASF contribution in 2014 and was a long time PMC member in the Apache Storm project. I'm betting more than 10 years of contribution on OSS.).
>
> Technically speaking, I'm effectively wasting my time to push the thing I think is the right way. I should be OK not to push this hard and it's not the best use of time from a vendor perspective. This is really just due diligence of the mistake, because I feel I need to do that.

This is code that will be carried for sometime, and regardless of what
you responded to, I asked about what *users* would be affected. I
think it's fine to let your response speak for itself.

Thanks
Andrew

>
>>
>> Without a direct response to this, I think this discussion should be
>> considered to just be what it is on it's face -- a solution to a
>> vendors mistake and should not be ported to OSS spark.
>>
>> Thanks
>> Andrew
>>
>> On Mon, Mar 10, 2025 at 8:34 PM Jungtaek Lim
>> <ka...@gmail.com> wrote:
>> >
>> > Please read through the explanation of how this impacts the OSS users in the other branch of this discussion. This happened in "Apache" Spark 3.5.4, and the migration logic has nothing to do with the vendor. This is primarily to not break users in "Apache" Spark 3.5.4 who are willing to upgrade directly to "Apache" Spark 4.0.0+.
>> >
>> > I'm not implying anything about compatibility between OSS and vendors. I would remind that the problematic config name is not problematic for the vendor, so the thing the vendor needs to do is just to alias incorrect config name and new config name and done. The vendor does not need to have such a migration code. This is just to do due diligence of mitigating the mistake we have made. Again, if the migration logic does not land to Apache Spark 4.0.x, only "Apache" Spark users will be impacted.
>> >
>> > On Tue, Mar 11, 2025 at 10:25 AM Andrew Melo <an...@gmail.com> wrote:
>> >>
>> >> Hello Jungtaek,
>> >>
>> >> I'm not implying that this improves the vendors life. I'm just not understanding the issue -- the downstream people started a stream with a config option that the upstream people don't want to carry. If the affected users are using the downstream fork (which is how they got the option), then why can't those same downstream users not keep using their vendor-provided downstream fork.
>> >>
>> >> To put it into less complicated words - presumably the people using the databricks.* configs were already using the databricks runtime. Why does Apache spark need to carry an extra migration patch when the users who would be affected are already using the Databricks fork? I don't see a situation where:
>> >>
>> >> A) legacy 3.5.x queries were using databricks-specific options
>> >> B) these users want to run the same queries in OSS spark today
>> >> C) the same people will not be using the databricks fork.
>> >>
>> >> This Venn diagram seems very small, and I don't think it justifies carrying migration code for that one sliver of users.
>> >>
>> >> Andrew
>> >>
>> >> On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <ka...@gmail.com> wrote:
>> >> >
>> >> > One thing I can correct immediately is, downstream does not have any impact at all from this. I believe I clarified that the config will not be modified by anyone, so downstream there is nothing to change. The problem is particular in OSS, downstream does not have any issue with this leak at all.
>> >> > (One thing to clarify, config itself will be removed in Spark 4.0.0. We only propose to retain migration logic.)
>> >> >
>> >> > I believe there is a huge misunderstanding - we are not proposing this migration logic to ease the vendor's life, no it's not. If I don't care about OSS, there is no incentive for me to propose this.
>> >> >
>> >> > I just wanted to do my best to remove any burden to users who are innocent with this problem. If there is no migration logic, users will be enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't it bad enough? Why do we have to let users be bugging while we can avoid it? The problem was introduced in the OSS community (I hope we don't blame ourselves with mistakes. We are human.) so it is up to us to resolve this properly. We don't have the right to break users' queries.
>> >> >
>> >> > On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com> wrote:
>> >> >>
>> >> >> Hello all
>> >> >>
>> >> >> As an outsider, I don't fully understand this discussion. This
>> >> >> particular configuration option "leaked" into the open-source Spark
>> >> >> distribution, and now there is a lot of discussion about how to
>> >> >> mitigate existing workloads. But: presumably the people who are
>> >> >> depending on this configuration flag are already using a downstream
>> >> >> (vendor-specific) fork, and a future update will similarly be
>> >> >> distributed by that downstream provider.
>> >> >>
>> >> >> Which people a) made a workflow using the vendor fork and b) want to
>> >> >> resume it in the OSS version of spark?
>> >> >>
>> >> >> It seems like the people who are affected by this will already be
>> >> >> using someone else's fork, and there's no need to carry this patch in
>> >> >> the mainline Spark code.
>> >> >>
>> >> >> For that reason, I believe the code should be dropped by OSS Spark,
>> >> >> and vendors who need to mitigate it can push the appropriate changes
>> >> >> to their downstreams.
>> >> >>
>> >> >> Thanks
>> >> >> Andrew

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

That said, if you guys understand the proposal better and have a preference
on one side, could you please participate in the VOTE thread?
https://lists.apache.org/thread/nm3p1zjcybdl0p0mc56t2rl92hb9837n

Specifically on this topic, I do think the input from users is very
important, especially if you are maintaining streaming queries in your
daily job.

Thanks!

On Tue, Mar 11, 2025 at 12:18 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Thanks for looking into the issue in depth. What you described is right.
>
> I also understand the concern why we keep the buggy behavior, but the QO
> issue is quite complicated and the most concerning part is that it's
> "selective". So if the query runs with QO's decision in "one way" in its
> lifecycle, it does not hit any issue, despite it going through the
> "fragile" path. If we don't make backward compatibility and enforce the
> fix, the existing query is at risk of "immediate failure on upgrade". I
> understand there are arguments about what is the right path, but the
> decision was made to not break the existing query from this fix. That's it.
>
> So the target workloads to help are every single streaming query that has
> ever "started" with Apache Spark 3.5.4. To help these workloads to not
> re-expose the nasty bug, we just need to have 4 lines of code. I understand
> someone may be concerned about ever having the vendor name in the codebase,
> but likewise Wenchen said, it had been there, and no one had been providing
> the exact ASF policy of this. If, ASF policy is well clearly stated that
> the codebase shouldn't have vendor name on it, there is no argument.
> Although I wonder whether this is realistic at all e.g. I can argue that we
> must remove the code comment of "Apple" Silicon to just "Silicon" or "M
> series" as it contains the vendor name. How about Meta? Should we prohibit
> Metadata be referenced with Meta? Is it really making everyone happy?
>
>
>
> On Tue, Mar 11, 2025 at 11:12 AM Adam Binford <ad...@gmail.com> wrote:
>
>> I was very confused about this as well but I think I understand it more
>> after reading through the PRs. Jungtaek let me know if this is correct,
>> maybe it will help others understand.
>>
>> There was a bug where streaming queries could prune parts of the query
>> that might have side effects, like stateful queries or metric collection.
>> This bug was fixed in OSS Spark 3.5.4, with a new config that was
>> accidentally added as "spark.databricks.sql". This config defaulted to
>> false, which is the value that "fixes" the bug. The config was added to
>> maintain the current behavior for existing structured streams. So for any
>> streams that were started in 3.5.3 or earlier, when they are restarted in
>> 3.5.4, the offset metadata log included a default value of "true" for the
>> "spark.databricks.sql" config, so as to keep the existing buggy behavior
>> for existing streams. Any new stream started in 3.5.4 would have this value
>> set to "false" in the offset metadata log because that is the default value
>> for the config. In 3.5.5 the config name was corrected to remove
>> "databricks".
>>
>> I'm not totally sure the reason for keeping the buggy behavior for
>> existing streams. It seems like eventually you will have to restart these
>> streams from scratch to fix the buggy behavior if you happen to hit the
>> edge case with the bug that causes problems. The case that keeping the
>> handling of the incorrect config name is trying to handle is:
>> - A new stream was started in 3.5.4 with the incorrect config name set to
>> "false" in the metadata log, fixing the bug
>> - This stream is restarted from the existing checkpoint in 4.0.0+
>> - Without the incorrect config handling logic, this restarted stream
>> would revert back to the "buggy" behavior until the stream is restarted
>> from a fresh checkpoint
>> - With the incorrect config handling, a stream restarted from an existing
>> checkpoint in 4.0.0 would maintain the bug fix while fixing the config name
>> in the metadata log
>>
>> Without the special handling of the incorrect config name, for all
>> intents and purposes this bug was fixed in 3.5.5 and later instead of
>> 3.5.4, so it doesn't seem like a huge deal either way if the code is
>> briefly maintained to help this specific edge case. Any stream started in
>> 3.5.3 would need to be restarted either in 3.5.4 or 3.5.5 and later to have
>> the bug fix, but if this bug is causing breaking problems for your query
>> you probably would need to restart it from scratch anyway.
>>
>> At the end of the day there's probably no reason not to include 4 lines
>> of code if it makes a few peoples' lives easier, not sure how many people
>> are affected by this bug.
>>
>> Adam
>>
>> On Mon, Mar 10, 2025 at 10:02 PM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> Replied inline
>>>
>>> On Tue, Mar 11, 2025 at 10:39 AM Andrew Melo <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi Jungtaek,
>>>>
>>>> I've read the discussion, which is why I replied with my questions
>>>> (which you neglected to answer). Your deflection and lack of response
>>>> to direct questions should be (IMO) disqualifying. So, again:
>>>>
>>>> To put it into less complicated words - presumably the people using
>>>> the databricks.* configs were already using the databricks runtime.
>>>> Why does Apache spark need to carry an extra migration patch when the
>>>> users who would be affected are already using the Databricks fork? I
>>>> don't see a situation where:
>>>>
>>>> A) legacy 3.5.x queries were using databricks-specific options
>>>> B) these users want to run the same queries in OSS spark today
>>>> C) the same people will not be using the databricks fork.
>>>>
>>>
>>> I don't understand why you couple this with Databricks Runtime.
>>>
>>> If I were pushing this to the case where the config was only released in
>>> Databricks Runtime and not yet to be released in Apache Spark, you are
>>> definitely right, although there are a bunch of other ways around to solve
>>> the issue on the vendor side, and I wouldn't choose this way which would be
>>> expected to have a lot of burden.
>>>
>>> This config was released to "Apache" Spark 3.5.4, so this is NO LONGER
>>> just a problem with vendor distribution. The breakage will happen even if
>>> someone does not even know about Databricks Runtime at all and keeps using
>>> Apache Spark. It just happens when users in Apache Spark 3.5.4 directly
>>> upgrade to Apache Spark 4.0.0+ (rather than upgrading once to Apache Spark
>>> 3.5.5+).
>>> (Again, from the vendor codebase, it is not a problem at all to have
>>> such a config and there is a much easier way to deal with it - migration
>>> logic is not needed at all.)
>>>
>>> I think this is a major misunderstanding that you see in this discussion
>>> to push vendor's code into OSS. Again, the migration logic is needed only
>>> by OSS. I can guarantee that I am not doing this for a vendor. (I made my
>>> first ASF contribution in 2014 and was a long time PMC member in the Apache
>>> Storm project. I'm betting more than 10 years of contribution on OSS.).
>>>
>>> Technically speaking, I'm effectively wasting my time to push the thing
>>> I think is the right way. I should be OK not to push this hard and it's not
>>> the best use of time from a vendor perspective. This is really just due
>>> diligence of the mistake, because I feel I need to do that.
>>>
>>>
>>>> Without a direct response to this, I think this discussion should be
>>>> considered to just be what it is on it's face -- a solution to a
>>>> vendors mistake and should not be ported to OSS spark.
>>>>
>>>> Thanks
>>>> Andrew
>>>>
>>>> On Mon, Mar 10, 2025 at 8:34 PM Jungtaek Lim
>>>> <ka...@gmail.com> wrote:
>>>> >
>>>> > Please read through the explanation of how this impacts the OSS users
>>>> in the other branch of this discussion. This happened in "Apache" Spark
>>>> 3.5.4, and the migration logic has nothing to do with the vendor. This is
>>>> primarily to not break users in "Apache" Spark 3.5.4 who are willing to
>>>> upgrade directly to "Apache" Spark 4.0.0+.
>>>> >
>>>> > I'm not implying anything about compatibility between OSS and
>>>> vendors. I would remind that the problematic config name is not problematic
>>>> for the vendor, so the thing the vendor needs to do is just to alias
>>>> incorrect config name and new config name and done. The vendor does not
>>>> need to have such a migration code. This is just to do due diligence of
>>>> mitigating the mistake we have made. Again, if the migration logic does not
>>>> land to Apache Spark 4.0.x, only "Apache" Spark users will be impacted.
>>>> >
>>>> > On Tue, Mar 11, 2025 at 10:25 AM Andrew Melo <an...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> Hello Jungtaek,
>>>> >>
>>>> >> I'm not implying that this improves the vendors life. I'm just not
>>>> understanding the issue -- the downstream people started a stream with a
>>>> config option that the upstream people don't want to carry. If the affected
>>>> users are using the downstream fork (which is how they got the option),
>>>> then why can't those same downstream users not keep using their
>>>> vendor-provided downstream fork.
>>>> >>
>>>> >> To put it into less complicated words - presumably the people using
>>>> the databricks.* configs were already using the databricks runtime. Why
>>>> does Apache spark need to carry an extra migration patch when the users who
>>>> would be affected are already using the Databricks fork? I don't see a
>>>> situation where:
>>>> >>
>>>> >> A) legacy 3.5.x queries were using databricks-specific options
>>>> >> B) these users want to run the same queries in OSS spark today
>>>> >> C) the same people will not be using the databricks fork.
>>>> >>
>>>> >> This Venn diagram seems very small, and I don't think it justifies
>>>> carrying migration code for that one sliver of users.
>>>> >>
>>>> >> Andrew
>>>> >>
>>>> >> On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>> >> >
>>>> >> > One thing I can correct immediately is, downstream does not have
>>>> any impact at all from this. I believe I clarified that the config will not
>>>> be modified by anyone, so downstream there is nothing to change. The
>>>> problem is particular in OSS, downstream does not have any issue with this
>>>> leak at all.
>>>> >> > (One thing to clarify, config itself will be removed in Spark
>>>> 4.0.0. We only propose to retain migration logic.)
>>>> >> >
>>>> >> > I believe there is a huge misunderstanding - we are not proposing
>>>> this migration logic to ease the vendor's life, no it's not. If I don't
>>>> care about OSS, there is no incentive for me to propose this.
>>>> >> >
>>>> >> > I just wanted to do my best to remove any burden to users who are
>>>> innocent with this problem. If there is no migration logic, users will be
>>>> enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't
>>>> it bad enough? Why do we have to let users be bugging while we can avoid
>>>> it? The problem was introduced in the OSS community (I hope we don't blame
>>>> ourselves with mistakes. We are human.) so it is up to us to resolve this
>>>> properly. We don't have the right to break users' queries.
>>>> >> >
>>>> >> > On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com>
>>>> wrote:
>>>> >> >>
>>>> >> >> Hello all
>>>> >> >>
>>>> >> >> As an outsider, I don't fully understand this discussion. This
>>>> >> >> particular configuration option "leaked" into the open-source
>>>> Spark
>>>> >> >> distribution, and now there is a lot of discussion about how to
>>>> >> >> mitigate existing workloads. But: presumably the people who are
>>>> >> >> depending on this configuration flag are already using a
>>>> downstream
>>>> >> >> (vendor-specific) fork, and a future update will similarly be
>>>> >> >> distributed by that downstream provider.
>>>> >> >>
>>>> >> >> Which people a) made a workflow using the vendor fork and b) want
>>>> to
>>>> >> >> resume it in the OSS version of spark?
>>>> >> >>
>>>> >> >> It seems like the people who are affected by this will already be
>>>> >> >> using someone else's fork, and there's no need to carry this
>>>> patch in
>>>> >> >> the mainline Spark code.
>>>> >> >>
>>>> >> >> For that reason, I believe the code should be dropped by OSS
>>>> Spark,
>>>> >> >> and vendors who need to mitigate it can push the appropriate
>>>> changes
>>>> >> >> to their downstreams.
>>>> >> >>
>>>> >> >> Thanks
>>>> >> >> Andrew
>>>>
>>>
>>
>> --
>> Adam Binford
>>
>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Thanks for looking into the issue in depth. What you described is right.

I also understand the concern why we keep the buggy behavior, but the QO
issue is quite complicated and the most concerning part is that it's
"selective". So if the query runs with QO's decision in "one way" in its
lifecycle, it does not hit any issue, despite it going through the
"fragile" path. If we don't make backward compatibility and enforce the
fix, the existing query is at risk of "immediate failure on upgrade". I
understand there are arguments about what is the right path, but the
decision was made to not break the existing query from this fix. That's it.

So the target workloads to help are every single streaming query that has
ever "started" with Apache Spark 3.5.4. To help these workloads to not
re-expose the nasty bug, we just need to have 4 lines of code. I understand
someone may be concerned about ever having the vendor name in the codebase,
but likewise Wenchen said, it had been there, and no one had been providing
the exact ASF policy of this. If, ASF policy is well clearly stated that
the codebase shouldn't have vendor name on it, there is no argument.
Although I wonder whether this is realistic at all e.g. I can argue that we
must remove the code comment of "Apple" Silicon to just "Silicon" or "M
series" as it contains the vendor name. How about Meta? Should we prohibit
Metadata be referenced with Meta? Is it really making everyone happy?



On Tue, Mar 11, 2025 at 11:12 AM Adam Binford <ad...@gmail.com> wrote:

> I was very confused about this as well but I think I understand it more
> after reading through the PRs. Jungtaek let me know if this is correct,
> maybe it will help others understand.
>
> There was a bug where streaming queries could prune parts of the query
> that might have side effects, like stateful queries or metric collection.
> This bug was fixed in OSS Spark 3.5.4, with a new config that was
> accidentally added as "spark.databricks.sql". This config defaulted to
> false, which is the value that "fixes" the bug. The config was added to
> maintain the current behavior for existing structured streams. So for any
> streams that were started in 3.5.3 or earlier, when they are restarted in
> 3.5.4, the offset metadata log included a default value of "true" for the
> "spark.databricks.sql" config, so as to keep the existing buggy behavior
> for existing streams. Any new stream started in 3.5.4 would have this value
> set to "false" in the offset metadata log because that is the default value
> for the config. In 3.5.5 the config name was corrected to remove
> "databricks".
>
> I'm not totally sure the reason for keeping the buggy behavior for
> existing streams. It seems like eventually you will have to restart these
> streams from scratch to fix the buggy behavior if you happen to hit the
> edge case with the bug that causes problems. The case that keeping the
> handling of the incorrect config name is trying to handle is:
> - A new stream was started in 3.5.4 with the incorrect config name set to
> "false" in the metadata log, fixing the bug
> - This stream is restarted from the existing checkpoint in 4.0.0+
> - Without the incorrect config handling logic, this restarted stream would
> revert back to the "buggy" behavior until the stream is restarted from a
> fresh checkpoint
> - With the incorrect config handling, a stream restarted from an existing
> checkpoint in 4.0.0 would maintain the bug fix while fixing the config name
> in the metadata log
>
> Without the special handling of the incorrect config name, for all intents
> and purposes this bug was fixed in 3.5.5 and later instead of 3.5.4, so it
> doesn't seem like a huge deal either way if the code is briefly maintained
> to help this specific edge case. Any stream started in 3.5.3 would need to
> be restarted either in 3.5.4 or 3.5.5 and later to have the bug fix, but if
> this bug is causing breaking problems for your query you probably would
> need to restart it from scratch anyway.
>
> At the end of the day there's probably no reason not to include 4 lines of
> code if it makes a few peoples' lives easier, not sure how many people are
> affected by this bug.
>
> Adam
>
> On Mon, Mar 10, 2025 at 10:02 PM Jungtaek Lim <
> kabhwan.opensource@gmail.com> wrote:
>
>> Replied inline
>>
>> On Tue, Mar 11, 2025 at 10:39 AM Andrew Melo <an...@gmail.com>
>> wrote:
>>
>>> Hi Jungtaek,
>>>
>>> I've read the discussion, which is why I replied with my questions
>>> (which you neglected to answer). Your deflection and lack of response
>>> to direct questions should be (IMO) disqualifying. So, again:
>>>
>>> To put it into less complicated words - presumably the people using
>>> the databricks.* configs were already using the databricks runtime.
>>> Why does Apache spark need to carry an extra migration patch when the
>>> users who would be affected are already using the Databricks fork? I
>>> don't see a situation where:
>>>
>>> A) legacy 3.5.x queries were using databricks-specific options
>>> B) these users want to run the same queries in OSS spark today
>>> C) the same people will not be using the databricks fork.
>>>
>>
>> I don't understand why you couple this with Databricks Runtime.
>>
>> If I were pushing this to the case where the config was only released in
>> Databricks Runtime and not yet to be released in Apache Spark, you are
>> definitely right, although there are a bunch of other ways around to solve
>> the issue on the vendor side, and I wouldn't choose this way which would be
>> expected to have a lot of burden.
>>
>> This config was released to "Apache" Spark 3.5.4, so this is NO LONGER
>> just a problem with vendor distribution. The breakage will happen even if
>> someone does not even know about Databricks Runtime at all and keeps using
>> Apache Spark. It just happens when users in Apache Spark 3.5.4 directly
>> upgrade to Apache Spark 4.0.0+ (rather than upgrading once to Apache Spark
>> 3.5.5+).
>> (Again, from the vendor codebase, it is not a problem at all to have such
>> a config and there is a much easier way to deal with it - migration logic
>> is not needed at all.)
>>
>> I think this is a major misunderstanding that you see in this discussion
>> to push vendor's code into OSS. Again, the migration logic is needed only
>> by OSS. I can guarantee that I am not doing this for a vendor. (I made my
>> first ASF contribution in 2014 and was a long time PMC member in the Apache
>> Storm project. I'm betting more than 10 years of contribution on OSS.).
>>
>> Technically speaking, I'm effectively wasting my time to push the thing I
>> think is the right way. I should be OK not to push this hard and it's not
>> the best use of time from a vendor perspective. This is really just due
>> diligence of the mistake, because I feel I need to do that.
>>
>>
>>> Without a direct response to this, I think this discussion should be
>>> considered to just be what it is on it's face -- a solution to a
>>> vendors mistake and should not be ported to OSS spark.
>>>
>>> Thanks
>>> Andrew
>>>
>>> On Mon, Mar 10, 2025 at 8:34 PM Jungtaek Lim
>>> <ka...@gmail.com> wrote:
>>> >
>>> > Please read through the explanation of how this impacts the OSS users
>>> in the other branch of this discussion. This happened in "Apache" Spark
>>> 3.5.4, and the migration logic has nothing to do with the vendor. This is
>>> primarily to not break users in "Apache" Spark 3.5.4 who are willing to
>>> upgrade directly to "Apache" Spark 4.0.0+.
>>> >
>>> > I'm not implying anything about compatibility between OSS and vendors.
>>> I would remind that the problematic config name is not problematic for the
>>> vendor, so the thing the vendor needs to do is just to alias incorrect
>>> config name and new config name and done. The vendor does not need to have
>>> such a migration code. This is just to do due diligence of mitigating the
>>> mistake we have made. Again, if the migration logic does not land to Apache
>>> Spark 4.0.x, only "Apache" Spark users will be impacted.
>>> >
>>> > On Tue, Mar 11, 2025 at 10:25 AM Andrew Melo <an...@gmail.com>
>>> wrote:
>>> >>
>>> >> Hello Jungtaek,
>>> >>
>>> >> I'm not implying that this improves the vendors life. I'm just not
>>> understanding the issue -- the downstream people started a stream with a
>>> config option that the upstream people don't want to carry. If the affected
>>> users are using the downstream fork (which is how they got the option),
>>> then why can't those same downstream users not keep using their
>>> vendor-provided downstream fork.
>>> >>
>>> >> To put it into less complicated words - presumably the people using
>>> the databricks.* configs were already using the databricks runtime. Why
>>> does Apache spark need to carry an extra migration patch when the users who
>>> would be affected are already using the Databricks fork? I don't see a
>>> situation where:
>>> >>
>>> >> A) legacy 3.5.x queries were using databricks-specific options
>>> >> B) these users want to run the same queries in OSS spark today
>>> >> C) the same people will not be using the databricks fork.
>>> >>
>>> >> This Venn diagram seems very small, and I don't think it justifies
>>> carrying migration code for that one sliver of users.
>>> >>
>>> >> Andrew
>>> >>
>>> >> On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>> >> >
>>> >> > One thing I can correct immediately is, downstream does not have
>>> any impact at all from this. I believe I clarified that the config will not
>>> be modified by anyone, so downstream there is nothing to change. The
>>> problem is particular in OSS, downstream does not have any issue with this
>>> leak at all.
>>> >> > (One thing to clarify, config itself will be removed in Spark
>>> 4.0.0. We only propose to retain migration logic.)
>>> >> >
>>> >> > I believe there is a huge misunderstanding - we are not proposing
>>> this migration logic to ease the vendor's life, no it's not. If I don't
>>> care about OSS, there is no incentive for me to propose this.
>>> >> >
>>> >> > I just wanted to do my best to remove any burden to users who are
>>> innocent with this problem. If there is no migration logic, users will be
>>> enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't
>>> it bad enough? Why do we have to let users be bugging while we can avoid
>>> it? The problem was introduced in the OSS community (I hope we don't blame
>>> ourselves with mistakes. We are human.) so it is up to us to resolve this
>>> properly. We don't have the right to break users' queries.
>>> >> >
>>> >> > On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com>
>>> wrote:
>>> >> >>
>>> >> >> Hello all
>>> >> >>
>>> >> >> As an outsider, I don't fully understand this discussion. This
>>> >> >> particular configuration option "leaked" into the open-source Spark
>>> >> >> distribution, and now there is a lot of discussion about how to
>>> >> >> mitigate existing workloads. But: presumably the people who are
>>> >> >> depending on this configuration flag are already using a downstream
>>> >> >> (vendor-specific) fork, and a future update will similarly be
>>> >> >> distributed by that downstream provider.
>>> >> >>
>>> >> >> Which people a) made a workflow using the vendor fork and b) want
>>> to
>>> >> >> resume it in the OSS version of spark?
>>> >> >>
>>> >> >> It seems like the people who are affected by this will already be
>>> >> >> using someone else's fork, and there's no need to carry this patch
>>> in
>>> >> >> the mainline Spark code.
>>> >> >>
>>> >> >> For that reason, I believe the code should be dropped by OSS Spark,
>>> >> >> and vendors who need to mitigate it can push the appropriate
>>> changes
>>> >> >> to their downstreams.
>>> >> >>
>>> >> >> Thanks
>>> >> >> Andrew
>>>
>>
>
> --
> Adam Binford
>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Adam Binford <ad...@gmail.com>]

I was very confused about this as well but I think I understand it more
after reading through the PRs. Jungtaek let me know if this is correct,
maybe it will help others understand.

There was a bug where streaming queries could prune parts of the query that
might have side effects, like stateful queries or metric collection. This
bug was fixed in OSS Spark 3.5.4, with a new config that was accidentally
added as "spark.databricks.sql". This config defaulted to false, which is
the value that "fixes" the bug. The config was added to maintain the
current behavior for existing structured streams. So for any streams that
were started in 3.5.3 or earlier, when they are restarted in 3.5.4, the
offset metadata log included a default value of "true" for the
"spark.databricks.sql" config, so as to keep the existing buggy behavior
for existing streams. Any new stream started in 3.5.4 would have this value
set to "false" in the offset metadata log because that is the default value
for the config. In 3.5.5 the config name was corrected to remove
"databricks".

I'm not totally sure the reason for keeping the buggy behavior for existing
streams. It seems like eventually you will have to restart these streams
from scratch to fix the buggy behavior if you happen to hit the edge case
with the bug that causes problems. The case that keeping the handling of
the incorrect config name is trying to handle is:
- A new stream was started in 3.5.4 with the incorrect config name set to
"false" in the metadata log, fixing the bug
- This stream is restarted from the existing checkpoint in 4.0.0+
- Without the incorrect config handling logic, this restarted stream would
revert back to the "buggy" behavior until the stream is restarted from a
fresh checkpoint
- With the incorrect config handling, a stream restarted from an existing
checkpoint in 4.0.0 would maintain the bug fix while fixing the config name
in the metadata log

Without the special handling of the incorrect config name, for all intents
and purposes this bug was fixed in 3.5.5 and later instead of 3.5.4, so it
doesn't seem like a huge deal either way if the code is briefly maintained
to help this specific edge case. Any stream started in 3.5.3 would need to
be restarted either in 3.5.4 or 3.5.5 and later to have the bug fix, but if
this bug is causing breaking problems for your query you probably would
need to restart it from scratch anyway.

At the end of the day there's probably no reason not to include 4 lines of
code if it makes a few peoples' lives easier, not sure how many people are
affected by this bug.

Adam

On Mon, Mar 10, 2025 at 10:02 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Replied inline
>
> On Tue, Mar 11, 2025 at 10:39 AM Andrew Melo <an...@gmail.com>
> wrote:
>
>> Hi Jungtaek,
>>
>> I've read the discussion, which is why I replied with my questions
>> (which you neglected to answer). Your deflection and lack of response
>> to direct questions should be (IMO) disqualifying. So, again:
>>
>> To put it into less complicated words - presumably the people using
>> the databricks.* configs were already using the databricks runtime.
>> Why does Apache spark need to carry an extra migration patch when the
>> users who would be affected are already using the Databricks fork? I
>> don't see a situation where:
>>
>> A) legacy 3.5.x queries were using databricks-specific options
>> B) these users want to run the same queries in OSS spark today
>> C) the same people will not be using the databricks fork.
>>
>
> I don't understand why you couple this with Databricks Runtime.
>
> If I were pushing this to the case where the config was only released in
> Databricks Runtime and not yet to be released in Apache Spark, you are
> definitely right, although there are a bunch of other ways around to solve
> the issue on the vendor side, and I wouldn't choose this way which would be
> expected to have a lot of burden.
>
> This config was released to "Apache" Spark 3.5.4, so this is NO LONGER
> just a problem with vendor distribution. The breakage will happen even if
> someone does not even know about Databricks Runtime at all and keeps using
> Apache Spark. It just happens when users in Apache Spark 3.5.4 directly
> upgrade to Apache Spark 4.0.0+ (rather than upgrading once to Apache Spark
> 3.5.5+).
> (Again, from the vendor codebase, it is not a problem at all to have such
> a config and there is a much easier way to deal with it - migration logic
> is not needed at all.)
>
> I think this is a major misunderstanding that you see in this discussion
> to push vendor's code into OSS. Again, the migration logic is needed only
> by OSS. I can guarantee that I am not doing this for a vendor. (I made my
> first ASF contribution in 2014 and was a long time PMC member in the Apache
> Storm project. I'm betting more than 10 years of contribution on OSS.).
>
> Technically speaking, I'm effectively wasting my time to push the thing I
> think is the right way. I should be OK not to push this hard and it's not
> the best use of time from a vendor perspective. This is really just due
> diligence of the mistake, because I feel I need to do that.
>
>
>> Without a direct response to this, I think this discussion should be
>> considered to just be what it is on it's face -- a solution to a
>> vendors mistake and should not be ported to OSS spark.
>>
>> Thanks
>> Andrew
>>
>> On Mon, Mar 10, 2025 at 8:34 PM Jungtaek Lim
>> <ka...@gmail.com> wrote:
>> >
>> > Please read through the explanation of how this impacts the OSS users
>> in the other branch of this discussion. This happened in "Apache" Spark
>> 3.5.4, and the migration logic has nothing to do with the vendor. This is
>> primarily to not break users in "Apache" Spark 3.5.4 who are willing to
>> upgrade directly to "Apache" Spark 4.0.0+.
>> >
>> > I'm not implying anything about compatibility between OSS and vendors.
>> I would remind that the problematic config name is not problematic for the
>> vendor, so the thing the vendor needs to do is just to alias incorrect
>> config name and new config name and done. The vendor does not need to have
>> such a migration code. This is just to do due diligence of mitigating the
>> mistake we have made. Again, if the migration logic does not land to Apache
>> Spark 4.0.x, only "Apache" Spark users will be impacted.
>> >
>> > On Tue, Mar 11, 2025 at 10:25 AM Andrew Melo <an...@gmail.com>
>> wrote:
>> >>
>> >> Hello Jungtaek,
>> >>
>> >> I'm not implying that this improves the vendors life. I'm just not
>> understanding the issue -- the downstream people started a stream with a
>> config option that the upstream people don't want to carry. If the affected
>> users are using the downstream fork (which is how they got the option),
>> then why can't those same downstream users not keep using their
>> vendor-provided downstream fork.
>> >>
>> >> To put it into less complicated words - presumably the people using
>> the databricks.* configs were already using the databricks runtime. Why
>> does Apache spark need to carry an extra migration patch when the users who
>> would be affected are already using the Databricks fork? I don't see a
>> situation where:
>> >>
>> >> A) legacy 3.5.x queries were using databricks-specific options
>> >> B) these users want to run the same queries in OSS spark today
>> >> C) the same people will not be using the databricks fork.
>> >>
>> >> This Venn diagram seems very small, and I don't think it justifies
>> carrying migration code for that one sliver of users.
>> >>
>> >> Andrew
>> >>
>> >> On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>> >> >
>> >> > One thing I can correct immediately is, downstream does not have any
>> impact at all from this. I believe I clarified that the config will not be
>> modified by anyone, so downstream there is nothing to change. The problem
>> is particular in OSS, downstream does not have any issue with this leak at
>> all.
>> >> > (One thing to clarify, config itself will be removed in Spark 4.0.0.
>> We only propose to retain migration logic.)
>> >> >
>> >> > I believe there is a huge misunderstanding - we are not proposing
>> this migration logic to ease the vendor's life, no it's not. If I don't
>> care about OSS, there is no incentive for me to propose this.
>> >> >
>> >> > I just wanted to do my best to remove any burden to users who are
>> innocent with this problem. If there is no migration logic, users will be
>> enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't
>> it bad enough? Why do we have to let users be bugging while we can avoid
>> it? The problem was introduced in the OSS community (I hope we don't blame
>> ourselves with mistakes. We are human.) so it is up to us to resolve this
>> properly. We don't have the right to break users' queries.
>> >> >
>> >> > On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com>
>> wrote:
>> >> >>
>> >> >> Hello all
>> >> >>
>> >> >> As an outsider, I don't fully understand this discussion. This
>> >> >> particular configuration option "leaked" into the open-source Spark
>> >> >> distribution, and now there is a lot of discussion about how to
>> >> >> mitigate existing workloads. But: presumably the people who are
>> >> >> depending on this configuration flag are already using a downstream
>> >> >> (vendor-specific) fork, and a future update will similarly be
>> >> >> distributed by that downstream provider.
>> >> >>
>> >> >> Which people a) made a workflow using the vendor fork and b) want to
>> >> >> resume it in the OSS version of spark?
>> >> >>
>> >> >> It seems like the people who are affected by this will already be
>> >> >> using someone else's fork, and there's no need to carry this patch
>> in
>> >> >> the mainline Spark code.
>> >> >>
>> >> >> For that reason, I believe the code should be dropped by OSS Spark,
>> >> >> and vendors who need to mitigate it can push the appropriate changes
>> >> >> to their downstreams.
>> >> >>
>> >> >> Thanks
>> >> >> Andrew
>>
>

-- 
Adam Binford

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Replied inline

On Tue, Mar 11, 2025 at 10:39 AM Andrew Melo <an...@gmail.com> wrote:

> Hi Jungtaek,
>
> I've read the discussion, which is why I replied with my questions
> (which you neglected to answer). Your deflection and lack of response
> to direct questions should be (IMO) disqualifying. So, again:
>
> To put it into less complicated words - presumably the people using
> the databricks.* configs were already using the databricks runtime.
> Why does Apache spark need to carry an extra migration patch when the
> users who would be affected are already using the Databricks fork? I
> don't see a situation where:
>
> A) legacy 3.5.x queries were using databricks-specific options
> B) these users want to run the same queries in OSS spark today
> C) the same people will not be using the databricks fork.
>

I don't understand why you couple this with Databricks Runtime.

If I were pushing this to the case where the config was only released in
Databricks Runtime and not yet to be released in Apache Spark, you are
definitely right, although there are a bunch of other ways around to solve
the issue on the vendor side, and I wouldn't choose this way which would be
expected to have a lot of burden.

This config was released to "Apache" Spark 3.5.4, so this is NO LONGER just
a problem with vendor distribution. The breakage will happen even if
someone does not even know about Databricks Runtime at all and keeps using
Apache Spark. It just happens when users in Apache Spark 3.5.4 directly
upgrade to Apache Spark 4.0.0+ (rather than upgrading once to Apache Spark
3.5.5+).
(Again, from the vendor codebase, it is not a problem at all to have such a
config and there is a much easier way to deal with it - migration logic is
not needed at all.)

I think this is a major misunderstanding that you see in this discussion
to push vendor's code into OSS. Again, the migration logic is needed only
by OSS. I can guarantee that I am not doing this for a vendor. (I made my
first ASF contribution in 2014 and was a long time PMC member in the Apache
Storm project. I'm betting more than 10 years of contribution on OSS.).

Technically speaking, I'm effectively wasting my time to push the thing I
think is the right way. I should be OK not to push this hard and it's not
the best use of time from a vendor perspective. This is really just due
diligence of the mistake, because I feel I need to do that.


> Without a direct response to this, I think this discussion should be
> considered to just be what it is on it's face -- a solution to a
> vendors mistake and should not be ported to OSS spark.
>
> Thanks
> Andrew
>
> On Mon, Mar 10, 2025 at 8:34 PM Jungtaek Lim
> <ka...@gmail.com> wrote:
> >
> > Please read through the explanation of how this impacts the OSS users in
> the other branch of this discussion. This happened in "Apache" Spark 3.5.4,
> and the migration logic has nothing to do with the vendor. This is
> primarily to not break users in "Apache" Spark 3.5.4 who are willing to
> upgrade directly to "Apache" Spark 4.0.0+.
> >
> > I'm not implying anything about compatibility between OSS and vendors. I
> would remind that the problematic config name is not problematic for the
> vendor, so the thing the vendor needs to do is just to alias incorrect
> config name and new config name and done. The vendor does not need to have
> such a migration code. This is just to do due diligence of mitigating the
> mistake we have made. Again, if the migration logic does not land to Apache
> Spark 4.0.x, only "Apache" Spark users will be impacted.
> >
> > On Tue, Mar 11, 2025 at 10:25 AM Andrew Melo <an...@gmail.com>
> wrote:
> >>
> >> Hello Jungtaek,
> >>
> >> I'm not implying that this improves the vendors life. I'm just not
> understanding the issue -- the downstream people started a stream with a
> config option that the upstream people don't want to carry. If the affected
> users are using the downstream fork (which is how they got the option),
> then why can't those same downstream users not keep using their
> vendor-provided downstream fork.
> >>
> >> To put it into less complicated words - presumably the people using the
> databricks.* configs were already using the databricks runtime. Why does
> Apache spark need to carry an extra migration patch when the users who
> would be affected are already using the Databricks fork? I don't see a
> situation where:
> >>
> >> A) legacy 3.5.x queries were using databricks-specific options
> >> B) these users want to run the same queries in OSS spark today
> >> C) the same people will not be using the databricks fork.
> >>
> >> This Venn diagram seems very small, and I don't think it justifies
> carrying migration code for that one sliver of users.
> >>
> >> Andrew
> >>
> >> On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <
> kabhwan.opensource@gmail.com> wrote:
> >> >
> >> > One thing I can correct immediately is, downstream does not have any
> impact at all from this. I believe I clarified that the config will not be
> modified by anyone, so downstream there is nothing to change. The problem
> is particular in OSS, downstream does not have any issue with this leak at
> all.
> >> > (One thing to clarify, config itself will be removed in Spark 4.0.0.
> We only propose to retain migration logic.)
> >> >
> >> > I believe there is a huge misunderstanding - we are not proposing
> this migration logic to ease the vendor's life, no it's not. If I don't
> care about OSS, there is no incentive for me to propose this.
> >> >
> >> > I just wanted to do my best to remove any burden to users who are
> innocent with this problem. If there is no migration logic, users will be
> enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't
> it bad enough? Why do we have to let users be bugging while we can avoid
> it? The problem was introduced in the OSS community (I hope we don't blame
> ourselves with mistakes. We are human.) so it is up to us to resolve this
> properly. We don't have the right to break users' queries.
> >> >
> >> > On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com>
> wrote:
> >> >>
> >> >> Hello all
> >> >>
> >> >> As an outsider, I don't fully understand this discussion. This
> >> >> particular configuration option "leaked" into the open-source Spark
> >> >> distribution, and now there is a lot of discussion about how to
> >> >> mitigate existing workloads. But: presumably the people who are
> >> >> depending on this configuration flag are already using a downstream
> >> >> (vendor-specific) fork, and a future update will similarly be
> >> >> distributed by that downstream provider.
> >> >>
> >> >> Which people a) made a workflow using the vendor fork and b) want to
> >> >> resume it in the OSS version of spark?
> >> >>
> >> >> It seems like the people who are affected by this will already be
> >> >> using someone else's fork, and there's no need to carry this patch in
> >> >> the mainline Spark code.
> >> >>
> >> >> For that reason, I believe the code should be dropped by OSS Spark,
> >> >> and vendors who need to mitigate it can push the appropriate changes
> >> >> to their downstreams.
> >> >>
> >> >> Thanks
> >> >> Andrew
>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Andrew Melo <an...@gmail.com>]

Hi Jungtaek,

I've read the discussion, which is why I replied with my questions
(which you neglected to answer). Your deflection and lack of response
to direct questions should be (IMO) disqualifying. So, again:

To put it into less complicated words - presumably the people using
the databricks.* configs were already using the databricks runtime.
Why does Apache spark need to carry an extra migration patch when the
users who would be affected are already using the Databricks fork? I
don't see a situation where:

A) legacy 3.5.x queries were using databricks-specific options
B) these users want to run the same queries in OSS spark today
C) the same people will not be using the databricks fork.

Without a direct response to this, I think this discussion should be
considered to just be what it is on it's face -- a solution to a
vendors mistake and should not be ported to OSS spark.

Thanks
Andrew

On Mon, Mar 10, 2025 at 8:34 PM Jungtaek Lim
<ka...@gmail.com> wrote:
>
> Please read through the explanation of how this impacts the OSS users in the other branch of this discussion. This happened in "Apache" Spark 3.5.4, and the migration logic has nothing to do with the vendor. This is primarily to not break users in "Apache" Spark 3.5.4 who are willing to upgrade directly to "Apache" Spark 4.0.0+.
>
> I'm not implying anything about compatibility between OSS and vendors. I would remind that the problematic config name is not problematic for the vendor, so the thing the vendor needs to do is just to alias incorrect config name and new config name and done. The vendor does not need to have such a migration code. This is just to do due diligence of mitigating the mistake we have made. Again, if the migration logic does not land to Apache Spark 4.0.x, only "Apache" Spark users will be impacted.
>
> On Tue, Mar 11, 2025 at 10:25 AM Andrew Melo <an...@gmail.com> wrote:
>>
>> Hello Jungtaek,
>>
>> I'm not implying that this improves the vendors life. I'm just not understanding the issue -- the downstream people started a stream with a config option that the upstream people don't want to carry. If the affected users are using the downstream fork (which is how they got the option), then why can't those same downstream users not keep using their vendor-provided downstream fork.
>>
>> To put it into less complicated words - presumably the people using the databricks.* configs were already using the databricks runtime. Why does Apache spark need to carry an extra migration patch when the users who would be affected are already using the Databricks fork? I don't see a situation where:
>>
>> A) legacy 3.5.x queries were using databricks-specific options
>> B) these users want to run the same queries in OSS spark today
>> C) the same people will not be using the databricks fork.
>>
>> This Venn diagram seems very small, and I don't think it justifies carrying migration code for that one sliver of users.
>>
>> Andrew
>>
>> On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <ka...@gmail.com> wrote:
>> >
>> > One thing I can correct immediately is, downstream does not have any impact at all from this. I believe I clarified that the config will not be modified by anyone, so downstream there is nothing to change. The problem is particular in OSS, downstream does not have any issue with this leak at all.
>> > (One thing to clarify, config itself will be removed in Spark 4.0.0. We only propose to retain migration logic.)
>> >
>> > I believe there is a huge misunderstanding - we are not proposing this migration logic to ease the vendor's life, no it's not. If I don't care about OSS, there is no incentive for me to propose this.
>> >
>> > I just wanted to do my best to remove any burden to users who are innocent with this problem. If there is no migration logic, users will be enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't it bad enough? Why do we have to let users be bugging while we can avoid it? The problem was introduced in the OSS community (I hope we don't blame ourselves with mistakes. We are human.) so it is up to us to resolve this properly. We don't have the right to break users' queries.
>> >
>> > On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com> wrote:
>> >>
>> >> Hello all
>> >>
>> >> As an outsider, I don't fully understand this discussion. This
>> >> particular configuration option "leaked" into the open-source Spark
>> >> distribution, and now there is a lot of discussion about how to
>> >> mitigate existing workloads. But: presumably the people who are
>> >> depending on this configuration flag are already using a downstream
>> >> (vendor-specific) fork, and a future update will similarly be
>> >> distributed by that downstream provider.
>> >>
>> >> Which people a) made a workflow using the vendor fork and b) want to
>> >> resume it in the OSS version of spark?
>> >>
>> >> It seems like the people who are affected by this will already be
>> >> using someone else's fork, and there's no need to carry this patch in
>> >> the mainline Spark code.
>> >>
>> >> For that reason, I believe the code should be dropped by OSS Spark,
>> >> and vendors who need to mitigate it can push the appropriate changes
>> >> to their downstreams.
>> >>
>> >> Thanks
>> >> Andrew

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Please read through the explanation of how this impacts the OSS users in
the other branch of this discussion. This happened in "Apache" Spark 3.5.4,
and the migration logic has nothing to do with the vendor. This is
primarily to not break users in "Apache" Spark 3.5.4 who are willing to
upgrade directly to "Apache" Spark 4.0.0+.

I'm not implying anything about compatibility between OSS and vendors. I
would remind that the problematic config name is not problematic for the
vendor, so the thing the vendor needs to do is just to alias incorrect
config name and new config name and done. The vendor does not need to have
such a migration code. This is just to do due diligence of mitigating the
mistake we have made. Again, if the migration logic does not land to Apache
Spark 4.0.x, only "Apache" Spark users will be impacted.

On Tue, Mar 11, 2025 at 10:25 AM Andrew Melo <an...@gmail.com> wrote:

> Hello Jungtaek,
>
> I'm not implying that this improves the vendors life. I'm just not
> understanding the issue -- the downstream people started a stream with a
> config option that the upstream people don't want to carry. If the affected
> users are using the downstream fork (which is how they got the option),
> then why can't those same downstream users not keep using their
> vendor-provided downstream fork.
>
> To put it into less complicated words - presumably the people using the
> databricks.* configs were already using the databricks runtime. Why does
> Apache spark need to carry an extra migration patch when the users who
> would be affected are already using the Databricks fork? I don't see a
> situation where:
>
> A) legacy 3.5.x queries were using databricks-specific options
> B) these users want to run the same queries in OSS spark today
> C) the same people will not be using the databricks fork.
>
> This Venn diagram seems very small, and I don't think it justifies
> carrying migration code for that one sliver of users.
>
> Andrew
>
> On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
> >
> > One thing I can correct immediately is, downstream does not have any
> impact at all from this. I believe I clarified that the config will not be
> modified by anyone, so downstream there is nothing to change. The problem
> is particular in OSS, downstream does not have any issue with this leak at
> all.
> > (One thing to clarify, config itself will be removed in Spark 4.0.0. We
> only propose to retain migration logic.)
> >
> > I believe there is a huge misunderstanding - we are not proposing this
> migration logic to ease the vendor's life, no it's not. If I don't care
> about OSS, there is no incentive for me to propose this.
> >
> > I just wanted to do my best to remove any burden to users who are
> innocent with this problem. If there is no migration logic, users will be
> enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't
> it bad enough? Why do we have to let users be bugging while we can avoid
> it? The problem was introduced in the OSS community (I hope we don't blame
> ourselves with mistakes. We are human.) so it is up to us to resolve this
> properly. We don't have the right to break users' queries.
> >
> > On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com>
> wrote:
> >>
> >> Hello all
> >>
> >> As an outsider, I don't fully understand this discussion. This
> >> particular configuration option "leaked" into the open-source Spark
> >> distribution, and now there is a lot of discussion about how to
> >> mitigate existing workloads. But: presumably the people who are
> >> depending on this configuration flag are already using a downstream
> >> (vendor-specific) fork, and a future update will similarly be
> >> distributed by that downstream provider.
> >>
> >> Which people a) made a workflow using the vendor fork and b) want to
> >> resume it in the OSS version of spark?
> >>
> >> It seems like the people who are affected by this will already be
> >> using someone else's fork, and there's no need to carry this patch in
> >> the mainline Spark code.
> >>
> >> For that reason, I believe the code should be dropped by OSS Spark,
> >> and vendors who need to mitigate it can push the appropriate changes
> >> to their downstreams.
> >>
> >> Thanks
> >> Andrew
>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Andrew Melo <an...@gmail.com>]

Hello Jungtaek,

I'm not implying that this improves the vendors life. I'm just not
understanding the issue -- the downstream people started a stream with a
config option that the upstream people don't want to carry. If the affected
users are using the downstream fork (which is how they got the option),
then why can't those same downstream users not keep using their
vendor-provided downstream fork.

To put it into less complicated words - presumably the people using the
databricks.* configs were already using the databricks runtime. Why does
Apache spark need to carry an extra migration patch when the users who
would be affected are already using the Databricks fork? I don't see a
situation where:

A) legacy 3.5.x queries were using databricks-specific options
B) these users want to run the same queries in OSS spark today
C) the same people will not be using the databricks fork.

This Venn diagram seems very small, and I don't think it justifies carrying
migration code for that one sliver of users.

Andrew

On Mon, Mar 10, 2025 at 5:29 PM Jungtaek Lim <ka...@gmail.com>
wrote:
>
> One thing I can correct immediately is, downstream does not have any
impact at all from this. I believe I clarified that the config will not be
modified by anyone, so downstream there is nothing to change. The problem
is particular in OSS, downstream does not have any issue with this leak at
all.
> (One thing to clarify, config itself will be removed in Spark 4.0.0. We
only propose to retain migration logic.)
>
> I believe there is a huge misunderstanding - we are not proposing this
migration logic to ease the vendor's life, no it's not. If I don't care
about OSS, there is no incentive for me to propose this.
>
> I just wanted to do my best to remove any burden to users who are
innocent with this problem. If there is no migration logic, users will be
enforced to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't
it bad enough? Why do we have to let users be bugging while we can avoid
it? The problem was introduced in the OSS community (I hope we don't blame
ourselves with mistakes. We are human.) so it is up to us to resolve this
properly. We don't have the right to break users' queries.
>
> On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com> wrote:
>>
>> Hello all
>>
>> As an outsider, I don't fully understand this discussion. This
>> particular configuration option "leaked" into the open-source Spark
>> distribution, and now there is a lot of discussion about how to
>> mitigate existing workloads. But: presumably the people who are
>> depending on this configuration flag are already using a downstream
>> (vendor-specific) fork, and a future update will similarly be
>> distributed by that downstream provider.
>>
>> Which people a) made a workflow using the vendor fork and b) want to
>> resume it in the OSS version of spark?
>>
>> It seems like the people who are affected by this will already be
>> using someone else's fork, and there's no need to carry this patch in
>> the mainline Spark code.
>>
>> For that reason, I believe the code should be dropped by OSS Spark,
>> and vendors who need to mitigate it can push the appropriate changes
>> to their downstreams.
>>
>> Thanks
>> Andrew

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

One thing I can correct immediately is, downstream does not have any impact
at all from this. I believe I clarified that the config will not be
modified by anyone, so downstream there is nothing to change. The problem
is particular in OSS, downstream does not have any issue with this leak at
all.
(One thing to clarify, config itself will be removed in Spark 4.0.0. We
only propose to retain migration logic.)

I believe there is a huge misunderstanding - we are not proposing this
migration logic to ease the vendor's life, no it's not. If I don't care
about OSS, there is no incentive for me to propose this.

I just wanted to do my best to remove any burden to users who are innocent
with this problem. If there is no migration logic, users will be enforced
to upgrade to Spark 3.5.5+ before upgrading to Spark 4.0.0+. Isn't it bad
enough? Why do we have to let users be bugging while we can avoid it? The
problem was introduced in the OSS community (I hope we don't blame
ourselves with mistakes. We are human.) so it is up to us to resolve this
properly. We don't have the right to break users' queries.

On Tue, Mar 11, 2025 at 7:13 AM Andrew Melo <an...@gmail.com> wrote:

> Hello all
>
> As an outsider, I don't fully understand this discussion. This
> particular configuration option "leaked" into the open-source Spark
> distribution, and now there is a lot of discussion about how to
> mitigate existing workloads. But: presumably the people who are
> depending on this configuration flag are already using a downstream
> (vendor-specific) fork, and a future update will similarly be
> distributed by that downstream provider.
>
> Which people a) made a workflow using the vendor fork and b) want to
> resume it in the OSS version of spark?
>
> It seems like the people who are affected by this will already be
> using someone else's fork, and there's no need to carry this patch in
> the mainline Spark code.
>
> For that reason, I believe the code should be dropped by OSS Spark,
> and vendors who need to mitigate it can push the appropriate changes
> to their downstreams.
>
> Thanks
> Andrew
>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Andrew Melo <an...@gmail.com>]

Hello all

As an outsider, I don't fully understand this discussion. This
particular configuration option "leaked" into the open-source Spark
distribution, and now there is a lot of discussion about how to
mitigate existing workloads. But: presumably the people who are
depending on this configuration flag are already using a downstream
(vendor-specific) fork, and a future update will similarly be
distributed by that downstream provider.

Which people a) made a workflow using the vendor fork and b) want to
resume it in the OSS version of spark?

It seems like the people who are affected by this will already be
using someone else's fork, and there's no need to carry this patch in
the mainline Spark code.

For that reason, I believe the code should be dropped by OSS Spark,
and vendors who need to mitigate it can push the appropriate changes
to their downstreams.

Thanks
Andrew

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Thanks for the input.

> From a quick glance it seems like the incorrect config would just be
ignored from the checkpoint, and the new config would just be applied with
the default value going forward.

That's not how it works.

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala#L128

This is the default value of config when the checkpoint does not contain
such a config in its offset log. The default value will be flipped. This is
to respect backward compatibility as the QO issue is "selectively" applied
and the bug would have already impacted the query.

That said, we fixed the QO issue in Spark 3.5.4, but for streaming queries
which started from Spark 3.5.4, upgrading to Spark 4.0.0+ will "disable"
the QO fix, exposing the query to the risk of sudden query failure and zero
way to fix the checkpoint.

Hope this clarifies your question.

Thanks,
Jungtaek Lim (HeartSaVioR)


On Mon, Mar 10, 2025 at 10:06 PM Adam Binford <ad...@gmail.com> wrote:

> As someone who has a lot of streams that have been restarted with 3.5.4, I
> would prefer not to have to restart everything with 3.5.5 but it's
> definitely doable. But my question is what is the actual behavior if the
> migration logic was removed? From a quick glance it seems like the
> incorrect config would just be ignored from the checkpoint, and the new
> config would just be applied with the default value going forward. Does
> something actually hard fail somewhere? Or is the only concern if someone
> overrode the default value for the incorrect config, that override wouldn't
> be kept going forward automatically?
>
> Adam
>
> On Fri, Mar 7, 2025 at 2:08 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> Thank you for leading the discussion, Jungtaek.
>>
>> +1 for voting because we couldn't build a unanimous consensus on this
>> specific topic.
>>
>> Thanks,
>> Dongjoon.
>>
>> On 2025/03/07 09:15:39 Jungtaek Lim wrote:
>> > I'll need to start VOTE to move this forward.
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
> --
> Adam Binford
>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Adam Binford <ad...@gmail.com>]

As someone who has a lot of streams that have been restarted with 3.5.4, I
would prefer not to have to restart everything with 3.5.5 but it's
definitely doable. But my question is what is the actual behavior if the
migration logic was removed? From a quick glance it seems like the
incorrect config would just be ignored from the checkpoint, and the new
config would just be applied with the default value going forward. Does
something actually hard fail somewhere? Or is the only concern if someone
overrode the default value for the incorrect config, that override wouldn't
be kept going forward automatically?

Adam

On Fri, Mar 7, 2025 at 2:08 PM Dongjoon Hyun <do...@apache.org> wrote:

> Thank you for leading the discussion, Jungtaek.
>
> +1 for voting because we couldn't build a unanimous consensus on this
> specific topic.
>
> Thanks,
> Dongjoon.
>
> On 2025/03/07 09:15:39 Jungtaek Lim wrote:
> > I'll need to start VOTE to move this forward.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

-- 
Adam Binford

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Dongjoon Hyun <do...@apache.org>]

Thank you for leading the discussion, Jungtaek.

+1 for voting because we couldn't build a unanimous consensus on this specific topic.

Thanks,
Dongjoon.

On 2025/03/07 09:15:39 Jungtaek Lim wrote:
> I'll need to start VOTE to move this forward.
> 

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

> Is the idea that the existence of the string 'databricks' in the code is
a problem? it is not.

This is the point where we have arguments. There is disagreement that this
is not just a string which I don't agree with, hence this discussion
thread. This is also why I want to see what is "evidence" (from ASF policy)
we have to take "extreme care" about this despite paying the cost of
surprising failure on existing streaming query in some users. This is also
why Wenchen pointed out we have been having vendor name in the codebase for
a while.

At least I can see support from a couple PMC members. If we are blocked for
the end of this week (including weekend), I'll need to start VOTE to move
this forward.

On Fri, Mar 7, 2025 at 5:41 PM Sean Owen <sr...@gmail.com> wrote:

> What is the problem with the existence of the migration logic? I
> understand not keeping the misnamed config. But the migration logic does no
> harm other than taking up a couple lines in the code, no?
> Unless someone offers any reason this is an issue... what are we even
> talking about.
>
> Is the idea that the existence of the string 'databricks' in the code is a
> problem? it is not.
>
> A simple test: Let us say we introduced a config called
> "spark.tubro.enabled" and fixed it to "spark.turbo.enabled". Would we be
> having the same conversation about retaining migration logic for way more
> than like 1 version? if not and the vendor name is the issue then I think
> that is the misunderstanding of... well I don't know what, still?
>
>
> On Fri, Mar 7, 2025 at 2:34 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Sean, it's about how long we keep the vendor name in the codebase (since
>> the migration logic has the problematic config name as a string) for users.
>>
>> We agreed in a previous discussion thread that "this is generally bad",
>> so we fixed the incorrect config name immediately in 3.5. Now it's just a
>> string somewhere in the codebase, so I really don't get why this has to be
>> immediately removed in 4.0.0, having a risk of breaking users' queries.
>>
>> It'd be awesome if you can confirm you understand the issue and still be
>> supportive to retain the migration logic as it is. Thanks!
>>
>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Sean Owen <sr...@gmail.com>]

What is the problem with the existence of the migration logic? I understand
not keeping the misnamed config. But the migration logic does no harm other
than taking up a couple lines in the code, no?
Unless someone offers any reason this is an issue... what are we even
talking about.

Is the idea that the existence of the string 'databricks' in the code is a
problem? it is not.

A simple test: Let us say we introduced a config called
"spark.tubro.enabled" and fixed it to "spark.turbo.enabled". Would we be
having the same conversation about retaining migration logic for way more
than like 1 version? if not and the vendor name is the issue then I think
that is the misunderstanding of... well I don't know what, still?


On Fri, Mar 7, 2025 at 2:34 AM Jungtaek Lim <ka...@gmail.com>
wrote:

> Sean, it's about how long we keep the vendor name in the codebase (since
> the migration logic has the problematic config name as a string) for users.
>
> We agreed in a previous discussion thread that "this is generally bad", so
> we fixed the incorrect config name immediately in 3.5. Now it's just a
> string somewhere in the codebase, so I really don't get why this has to be
> immediately removed in 4.0.0, having a risk of breaking users' queries.
>
> It'd be awesome if you can confirm you understand the issue and still be
> supportive to retain the migration logic as it is. Thanks!
>
>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Sean, it's about how long we keep the vendor name in the codebase (since
the migration logic has the problematic config name as a string) for users.

We agreed in a previous discussion thread that "this is generally bad", so
we fixed the incorrect config name immediately in 3.5. Now it's just a
string somewhere in the codebase, so I really don't get why this has to be
immediately removed in 4.0.0, having a risk of breaking users' queries.

It'd be awesome if you can confirm you understand the issue and still be
supportive to retain the migration logic as it is. Thanks!

On Fri, Mar 7, 2025 at 5:24 PM Sean Owen <sr...@gmail.com> wrote:

> I don't understand the problem with keeping migration logic in for a long
> time, just in case. Who cares, it's some bit of check buried somewhere in
> the streaming code, much like deprecation warnings.
> There is not somehow an ASF policy compelling the removal of such logic;
> you are not _required_ to remove deprecated code at major versions or
> something.
> Not removing at version 4 doesn't mean you can't remove at version 5, or
> even 4.2 or whatever.
>
> If there's no valid technical or policy reason offered to prohibit the
> migration logic, and there's a pretty clear technical reason to keep it
> (not breaking queries), then I just don't see what the question is. Keep it.
>
> On Fri, Mar 7, 2025 at 1:47 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> I have to cast -1 (despite non-binding) for every single RC for Spark
>> 4.0.0 till this is settled, since I don't agree with the current status
>> (Dongjoon's proposal is as-is).
>>
>> On the other hand, I want to unblock this and stop bugging the RC phase.
>> Again, I could be only persuaded if this is mandatory with ASF. There are a
>> couple ways to confirm:
>>
>> 1) Escalate this to relevant mailing list in ASF (We will need to figure
>> out "where", since I haven't had such a thing to be discussed beyond the
>> project)
>> 2) Figure out the ASF doc containing the policy of vendor name in the
>> codebase, and what action PMC members/committers needs to take.
>>
>> That should contain evidence that we should pay "any" cost to fix that
>> e.g. breaking users' queries. It's not justified if it simply says "remove
>> it".
>>
>> I don't know there is 2), so if anyone can find this, that would be
>> awesome. If anyone can help to figure out where to post to seek for voice
>> of 1), that would be also awesome.
>>
>> We can't wait forever with the above, so if no one has any strong voice
>> on this, I'm going to start VOTE with my proposal early next week, starting
>> with retaining migration code in 4.0.x, and 4.1.x if VOTE for 4.0.x
>> succeeds.
>>
>> On Thu, Mar 6, 2025 at 5:54 AM Jungtaek Lim <ka...@gmail.com>
>> wrote:
>>
>>> I think it is how to handle the deprecation and removal.
>>>
>>> If we leave the migration path for Spark 4.1.x, it will take more than
>>> "1 year of upgrade path" to be successful. From our release cadence, Spark
>>> 4.2.0 would probably be released March next year or later. And Spark 3.5.4
>>> was released in December last year. Spark 4.0.x may not be very long, but
>>> still provide 6+ months of upgrade path.
>>>
>>> I'm not saying we should leave it forever. I'm saying we should try to
>>> reduce the probability, likewise how projects handle the deprecation and
>>> removal while trying to minimize the impact. I see you are predicting the
>>> target to be not much, but that doesn't justify that we are free to do
>>> nothing and make them bugged and dissatisfied with the project.
>>>
>>> I see this is projected with "security fix" when we talk about severity,
>>> but security fix does not restrict upgrading path, so what we are about to
>>> do is much worse than that. I'm trying to make this a lot less worse.
>>>
>>> I'm doing my best to care about users. Upgrading is not just a "one
>>> click", even for bugfix versions.
>>>
>>> On Thu, Mar 6, 2025 at 1:56 AM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> Let me reformulate your suggestions and my interpretation.
>>>>
>>>> Option 1 "Adding back `spark.databricks.*` in Spark codebase and keep
>>>> forever"
>>>>
>>>> If we follow the proposed logic and reasoning, it means there is no
>>>> safe version to remove that configuration because Apache Spark 3.5.4 users
>>>> can jump to any future releases like Spark 4.1.0, 4.2.0, and 5.0.0
>>>> technically. In other words, we cannot remove that logic forever.
>>>>
>>>> That's the reason why we couldn't make an agreement so far.
>>>>
>>>> Option 2 is simply adding a sentence (or more accurate one) for Spark
>>>> 3.5.4 into the Spark 4.0.0 guideline because all other Spark versions
>>>> (except 3.5.4) are not contaminated by `spark.databricks.*` conf.
>>>>
>>>> "For Spark 3.5.4 streaming jobs, if you want to migrate the existing
>>>> running jobs, we need to upgrade them to Spark 3.5.5+ before upgrading
>>>> Spark 4.0"
>>>>
>>>> Dongjoon.
>>>>
>>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Sean Owen <sr...@gmail.com>]

I don't understand the problem with keeping migration logic in for a long
time, just in case. Who cares, it's some bit of check buried somewhere in
the streaming code, much like deprecation warnings.
There is not somehow an ASF policy compelling the removal of such logic;
you are not _required_ to remove deprecated code at major versions or
something.
Not removing at version 4 doesn't mean you can't remove at version 5, or
even 4.2 or whatever.

If there's no valid technical or policy reason offered to prohibit the
migration logic, and there's a pretty clear technical reason to keep it
(not breaking queries), then I just don't see what the question is. Keep it.

On Fri, Mar 7, 2025 at 1:47 AM Jungtaek Lim <ka...@gmail.com>
wrote:

> I have to cast -1 (despite non-binding) for every single RC for Spark
> 4.0.0 till this is settled, since I don't agree with the current status
> (Dongjoon's proposal is as-is).
>
> On the other hand, I want to unblock this and stop bugging the RC phase.
> Again, I could be only persuaded if this is mandatory with ASF. There are a
> couple ways to confirm:
>
> 1) Escalate this to relevant mailing list in ASF (We will need to figure
> out "where", since I haven't had such a thing to be discussed beyond the
> project)
> 2) Figure out the ASF doc containing the policy of vendor name in the
> codebase, and what action PMC members/committers needs to take.
>
> That should contain evidence that we should pay "any" cost to fix that
> e.g. breaking users' queries. It's not justified if it simply says "remove
> it".
>
> I don't know there is 2), so if anyone can find this, that would be
> awesome. If anyone can help to figure out where to post to seek for voice
> of 1), that would be also awesome.
>
> We can't wait forever with the above, so if no one has any strong voice on
> this, I'm going to start VOTE with my proposal early next week, starting
> with retaining migration code in 4.0.x, and 4.1.x if VOTE for 4.0.x
> succeeds.
>
> On Thu, Mar 6, 2025 at 5:54 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> I think it is how to handle the deprecation and removal.
>>
>> If we leave the migration path for Spark 4.1.x, it will take more than "1
>> year of upgrade path" to be successful. From our release cadence, Spark
>> 4.2.0 would probably be released March next year or later. And Spark 3.5.4
>> was released in December last year. Spark 4.0.x may not be very long, but
>> still provide 6+ months of upgrade path.
>>
>> I'm not saying we should leave it forever. I'm saying we should try to
>> reduce the probability, likewise how projects handle the deprecation and
>> removal while trying to minimize the impact. I see you are predicting the
>> target to be not much, but that doesn't justify that we are free to do
>> nothing and make them bugged and dissatisfied with the project.
>>
>> I see this is projected with "security fix" when we talk about severity,
>> but security fix does not restrict upgrading path, so what we are about to
>> do is much worse than that. I'm trying to make this a lot less worse.
>>
>> I'm doing my best to care about users. Upgrading is not just a "one
>> click", even for bugfix versions.
>>
>> On Thu, Mar 6, 2025 at 1:56 AM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Let me reformulate your suggestions and my interpretation.
>>>
>>> Option 1 "Adding back `spark.databricks.*` in Spark codebase and keep
>>> forever"
>>>
>>> If we follow the proposed logic and reasoning, it means there is no safe
>>> version to remove that configuration because Apache Spark 3.5.4 users can
>>> jump to any future releases like Spark 4.1.0, 4.2.0, and 5.0.0 technically.
>>> In other words, we cannot remove that logic forever.
>>>
>>> That's the reason why we couldn't make an agreement so far.
>>>
>>> Option 2 is simply adding a sentence (or more accurate one) for Spark
>>> 3.5.4 into the Spark 4.0.0 guideline because all other Spark versions
>>> (except 3.5.4) are not contaminated by `spark.databricks.*` conf.
>>>
>>> "For Spark 3.5.4 streaming jobs, if you want to migrate the existing
>>> running jobs, we need to upgrade them to Spark 3.5.5+ before upgrading
>>> Spark 4.0"
>>>
>>> Dongjoon.
>>>
>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

I have to cast -1 (despite non-binding) for every single RC for Spark 4.0.0
till this is settled, since I don't agree with the current status
(Dongjoon's proposal is as-is).

On the other hand, I want to unblock this and stop bugging the RC phase.
Again, I could be only persuaded if this is mandatory with ASF. There are a
couple ways to confirm:

1) Escalate this to relevant mailing list in ASF (We will need to figure
out "where", since I haven't had such a thing to be discussed beyond the
project)
2) Figure out the ASF doc containing the policy of vendor name in the
codebase, and what action PMC members/committers needs to take.

That should contain evidence that we should pay "any" cost to fix that e.g.
breaking users' queries. It's not justified if it simply says "remove it".

I don't know there is 2), so if anyone can find this, that would be
awesome. If anyone can help to figure out where to post to seek for voice
of 1), that would be also awesome.

We can't wait forever with the above, so if no one has any strong voice on
this, I'm going to start VOTE with my proposal early next week, starting
with retaining migration code in 4.0.x, and 4.1.x if VOTE for 4.0.x
succeeds.

On Thu, Mar 6, 2025 at 5:54 AM Jungtaek Lim <ka...@gmail.com>
wrote:

> I think it is how to handle the deprecation and removal.
>
> If we leave the migration path for Spark 4.1.x, it will take more than "1
> year of upgrade path" to be successful. From our release cadence, Spark
> 4.2.0 would probably be released March next year or later. And Spark 3.5.4
> was released in December last year. Spark 4.0.x may not be very long, but
> still provide 6+ months of upgrade path.
>
> I'm not saying we should leave it forever. I'm saying we should try to
> reduce the probability, likewise how projects handle the deprecation and
> removal while trying to minimize the impact. I see you are predicting the
> target to be not much, but that doesn't justify that we are free to do
> nothing and make them bugged and dissatisfied with the project.
>
> I see this is projected with "security fix" when we talk about severity,
> but security fix does not restrict upgrading path, so what we are about to
> do is much worse than that. I'm trying to make this a lot less worse.
>
> I'm doing my best to care about users. Upgrading is not just a "one
> click", even for bugfix versions.
>
> On Thu, Mar 6, 2025 at 1:56 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Let me reformulate your suggestions and my interpretation.
>>
>> Option 1 "Adding back `spark.databricks.*` in Spark codebase and keep
>> forever"
>>
>> If we follow the proposed logic and reasoning, it means there is no safe
>> version to remove that configuration because Apache Spark 3.5.4 users can
>> jump to any future releases like Spark 4.1.0, 4.2.0, and 5.0.0 technically.
>> In other words, we cannot remove that logic forever.
>>
>> That's the reason why we couldn't make an agreement so far.
>>
>> Option 2 is simply adding a sentence (or more accurate one) for Spark
>> 3.5.4 into the Spark 4.0.0 guideline because all other Spark versions
>> (except 3.5.4) are not contaminated by `spark.databricks.*` conf.
>>
>> "For Spark 3.5.4 streaming jobs, if you want to migrate the existing
>> running jobs, we need to upgrade them to Spark 3.5.5+ before upgrading
>> Spark 4.0"
>>
>> Dongjoon.
>>
>>
>> On Tue, Mar 4, 2025 at 11:11 PM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> Let's not start with VOTE right now, but let me make clear about options
>>> and pros/cons for the option, so that people can choose one over another.
>>>
>>> Option 1 (Current proposal): retain migration logic for Spark 4.0 (and
>>> maybe more minor versions, up to decision) which contains the problematic
>>> config as "string".
>>>
>>> Pros: We can avoid breaking users' queries in any path of upgrade, as
>>> long as we retain the migration logic. For example, we can support
>>> upgrading the streaming query which ever ran in Spark 3.5.4 to Spark 4.0.x
>>> as long as we decide to retain the migration logic for Spark 4.0. Spark
>>> 4.1.x, Spark 4.2.x, etc. as long as we retain the migration path longer.
>>> Cons: We retain the concerned config name in the codebase, though it's a
>>> string and users can never set it.
>>>
>>> Option 2 (Dongjoon's proposal): do not bring the migration logic in
>>> Spark 4.0 and force users to run existing streaming query with Spark 3.5.5+
>>> before upgrading to Spark 4.0.0+.
>>>
>>> Pros: We stop retaining the concerned config name in the codebase.
>>> Cons: Upgrading directly from Spark 3.5.4 to Spark 4.0+ will be missing
>>> the critical QO fix, which can lead to a "broken" checkpoint. If the
>>> checkpoint is broken, there is no way to restore and users have to restart
>>> the query from scratch. Since the target workload is stateful, in the worst
>>> case, the query has to start from the earliest data.
>>>
>>> I would only agree about the severity if the ASF project had a case of
>>> vendor name in codebase and it was decided to pay whatever cost to fix the
>>> case. I'm happy to be corrected if we have the doc in ASF explicitly
>>> mentioning the case and action item.
>>>
>>> On Wed, Mar 5, 2025 at 3:51 PM Wenchen Fan <cl...@gmail.com> wrote:
>>>
>>>> Shall we open an official vote for it? We can put more details on it so
>>>> that people can vote:
>>>> 1. how does it break user workloads without this migration code?
>>>> 2. what is the Apache policy for leaked vendor names in the codebase? I
>>>> think this is not the only one, we also mentioned
>>>> `com.databricks.spark.csv` in
>>>> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L621C8-L621C32
>>>>
>>>> On Wed, Mar 5, 2025 at 2:40 PM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>>
>>>>> One major question: How do you believe that we can enforce users on
>>>>> upgrading path? I have seen a bunch of cases where users upgrade 2-3 minor
>>>>> versions at once. Do you really believe we can just break their query?
>>>>> What's the data backing up your claim?
>>>>>
>>>>> I think we agree to disagree. I really don't want "users" to get into
>>>>> situations just because of us. It's regardless of who made the mistake -
>>>>> it's about what's the proper mitigation for this, and I do not believe
>>>>> enforcing users to upgrade to Spark 3.5.8+ before upgrading Spark 4.0 is a
>>>>> valid approach.
>>>>>
>>>>> If I could vote for your alternative option, I'm -1 for it.
>>>>>
>>>>> On Wed, Mar 5, 2025 at 3:29 PM Dongjoon Hyun <do...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Technically, there is no agreement here. In other words, we have the
>>>>>> same situation with the initial discussion thread where we couldn't build a
>>>>>> community consensus on this.
>>>>>>
>>>>>> > I will consider this as "lazy consensus" if there are no objections
>>>>>> > for 3 days from initiation of the thread.
>>>>>>
>>>>>> If you need an explicit veto, here is mine, -1, because I don't think
>>>>>> that's just a string.
>>>>>>
>>>>>> > the problematic config is just a "string",
>>>>>>
>>>>>> To be clear, as I proposed both in the PR comments and initial
>>>>>> discussion thread, I believe we had better keep the AS-IS `master` and
>>>>>> `branch-4.0` and recommend to upgrade to the latest version of Apache Spark
>>>>>> 3.5.x first before upgrading to Spark 4.
>>>>>>
>>>>>> Sincerely,
>>>>>> Dongjoon.
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 4, 2025 at 8:37 PM Jungtaek Lim <
>>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>>
>>>>>>> Bumping on this. Again, this is a blocker for Spark 4.0.0. I will
>>>>>>> consider this as "lazy consensus" if there are no objections for 3 days
>>>>>>> from initiation of the thread.
>>>>>>>
>>>>>>> On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <
>>>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi dev,
>>>>>>>>
>>>>>>>> This is a spin-up of the original thread "Deprecating and banning
>>>>>>>> `spark.databricks.*` config from Apache Spark repository". (link
>>>>>>>> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>>>>>>>>
>>>>>>>> From the original thread, we decided to deprecate the config in
>>>>>>>> Spark 3.5.5 and remove the config in Spark 4.0.0. That thread did not
>>>>>>>> decide one thing, about smooth migration logic.
>>>>>>>>
>>>>>>>> We "persist" the config into offset log for streaming query since
>>>>>>>> the value of the config must be consistent during the lifecycle of the
>>>>>>>> query. This means, the problematic config is already persisted for
>>>>>>>> streaming query which ever ran with Spark 3.5.4.
>>>>>>>>
>>>>>>>> For the migration logic, we re-assign the value of the problematic
>>>>>>>> config to the new config. This happens when the query is restarted, and it
>>>>>>>> will be reflected into an offset log for "newer batch" so after a couple
>>>>>>>> new microbatches the migration logic isn't needed. This migration logic is
>>>>>>>> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
>>>>>>>> couple microbatches, it will be mitigated.
>>>>>>>>
>>>>>>>> But I would say that there will always be a case that users just
>>>>>>>> bump the minor/major version without following all the bugfix versions. I
>>>>>>>> think it is still dangerous to remove the migration logic in Spark 4.0.0
>>>>>>>> (and probably Spark 4.1.0, depending on the discussion). From the migration
>>>>>>>> logic, the problematic config is just a "string", and users wouldn't be
>>>>>>>> able to set the value with the problematic config name. We don't document
>>>>>>>> this, as it'll be done automatically.
>>>>>>>>
>>>>>>>> That said, I'd propose to have migration logic for Spark 4.0
>>>>>>>> version line (at minimum, 4.1 is debatable). This will give a safer and
>>>>>>>> less burden migration path for users with just retaining a problematic
>>>>>>>> "string" (again, not a config).
>>>>>>>>
>>>>>>>> I'd love to hear the community's voice on this. I'd like to remind
>>>>>>>> you, this is a blocker for Spark 4.0.0.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>>
>>>>>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

I think it is how to handle the deprecation and removal.

If we leave the migration path for Spark 4.1.x, it will take more than "1
year of upgrade path" to be successful. From our release cadence, Spark
4.2.0 would probably be released March next year or later. And Spark 3.5.4
was released in December last year. Spark 4.0.x may not be very long, but
still provide 6+ months of upgrade path.

I'm not saying we should leave it forever. I'm saying we should try to
reduce the probability, likewise how projects handle the deprecation and
removal while trying to minimize the impact. I see you are predicting the
target to be not much, but that doesn't justify that we are free to do
nothing and make them bugged and dissatisfied with the project.

I see this is projected with "security fix" when we talk about severity,
but security fix does not restrict upgrading path, so what we are about to
do is much worse than that. I'm trying to make this a lot less worse.

I'm doing my best to care about users. Upgrading is not just a "one click",
even for bugfix versions.

On Thu, Mar 6, 2025 at 1:56 AM Dongjoon Hyun <do...@gmail.com>
wrote:

> Let me reformulate your suggestions and my interpretation.
>
> Option 1 "Adding back `spark.databricks.*` in Spark codebase and keep
> forever"
>
> If we follow the proposed logic and reasoning, it means there is no safe
> version to remove that configuration because Apache Spark 3.5.4 users can
> jump to any future releases like Spark 4.1.0, 4.2.0, and 5.0.0 technically.
> In other words, we cannot remove that logic forever.
>
> That's the reason why we couldn't make an agreement so far.
>
> Option 2 is simply adding a sentence (or more accurate one) for Spark
> 3.5.4 into the Spark 4.0.0 guideline because all other Spark versions
> (except 3.5.4) are not contaminated by `spark.databricks.*` conf.
>
> "For Spark 3.5.4 streaming jobs, if you want to migrate the existing
> running jobs, we need to upgrade them to Spark 3.5.5+ before upgrading
> Spark 4.0"
>
> Dongjoon.
>
>
> On Tue, Mar 4, 2025 at 11:11 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Let's not start with VOTE right now, but let me make clear about options
>> and pros/cons for the option, so that people can choose one over another.
>>
>> Option 1 (Current proposal): retain migration logic for Spark 4.0 (and
>> maybe more minor versions, up to decision) which contains the problematic
>> config as "string".
>>
>> Pros: We can avoid breaking users' queries in any path of upgrade, as
>> long as we retain the migration logic. For example, we can support
>> upgrading the streaming query which ever ran in Spark 3.5.4 to Spark 4.0.x
>> as long as we decide to retain the migration logic for Spark 4.0. Spark
>> 4.1.x, Spark 4.2.x, etc. as long as we retain the migration path longer.
>> Cons: We retain the concerned config name in the codebase, though it's a
>> string and users can never set it.
>>
>> Option 2 (Dongjoon's proposal): do not bring the migration logic in Spark
>> 4.0 and force users to run existing streaming query with Spark 3.5.5+
>> before upgrading to Spark 4.0.0+.
>>
>> Pros: We stop retaining the concerned config name in the codebase.
>> Cons: Upgrading directly from Spark 3.5.4 to Spark 4.0+ will be missing
>> the critical QO fix, which can lead to a "broken" checkpoint. If the
>> checkpoint is broken, there is no way to restore and users have to restart
>> the query from scratch. Since the target workload is stateful, in the worst
>> case, the query has to start from the earliest data.
>>
>> I would only agree about the severity if the ASF project had a case of
>> vendor name in codebase and it was decided to pay whatever cost to fix the
>> case. I'm happy to be corrected if we have the doc in ASF explicitly
>> mentioning the case and action item.
>>
>> On Wed, Mar 5, 2025 at 3:51 PM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> Shall we open an official vote for it? We can put more details on it so
>>> that people can vote:
>>> 1. how does it break user workloads without this migration code?
>>> 2. what is the Apache policy for leaked vendor names in the codebase? I
>>> think this is not the only one, we also mentioned
>>> `com.databricks.spark.csv` in
>>> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L621C8-L621C32
>>>
>>> On Wed, Mar 5, 2025 at 2:40 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> One major question: How do you believe that we can enforce users on
>>>> upgrading path? I have seen a bunch of cases where users upgrade 2-3 minor
>>>> versions at once. Do you really believe we can just break their query?
>>>> What's the data backing up your claim?
>>>>
>>>> I think we agree to disagree. I really don't want "users" to get into
>>>> situations just because of us. It's regardless of who made the mistake -
>>>> it's about what's the proper mitigation for this, and I do not believe
>>>> enforcing users to upgrade to Spark 3.5.8+ before upgrading Spark 4.0 is a
>>>> valid approach.
>>>>
>>>> If I could vote for your alternative option, I'm -1 for it.
>>>>
>>>> On Wed, Mar 5, 2025 at 3:29 PM Dongjoon Hyun <do...@gmail.com>
>>>> wrote:
>>>>
>>>>> Technically, there is no agreement here. In other words, we have the
>>>>> same situation with the initial discussion thread where we couldn't build a
>>>>> community consensus on this.
>>>>>
>>>>> > I will consider this as "lazy consensus" if there are no objections
>>>>> > for 3 days from initiation of the thread.
>>>>>
>>>>> If you need an explicit veto, here is mine, -1, because I don't think
>>>>> that's just a string.
>>>>>
>>>>> > the problematic config is just a "string",
>>>>>
>>>>> To be clear, as I proposed both in the PR comments and initial
>>>>> discussion thread, I believe we had better keep the AS-IS `master` and
>>>>> `branch-4.0` and recommend to upgrade to the latest version of Apache Spark
>>>>> 3.5.x first before upgrading to Spark 4.
>>>>>
>>>>> Sincerely,
>>>>> Dongjoon.
>>>>>
>>>>>
>>>>> On Tue, Mar 4, 2025 at 8:37 PM Jungtaek Lim <
>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>
>>>>>> Bumping on this. Again, this is a blocker for Spark 4.0.0. I will
>>>>>> consider this as "lazy consensus" if there are no objections for 3 days
>>>>>> from initiation of the thread.
>>>>>>
>>>>>> On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <
>>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>>
>>>>>>> Hi dev,
>>>>>>>
>>>>>>> This is a spin-up of the original thread "Deprecating and banning
>>>>>>> `spark.databricks.*` config from Apache Spark repository". (link
>>>>>>> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>>>>>>>
>>>>>>> From the original thread, we decided to deprecate the config in
>>>>>>> Spark 3.5.5 and remove the config in Spark 4.0.0. That thread did not
>>>>>>> decide one thing, about smooth migration logic.
>>>>>>>
>>>>>>> We "persist" the config into offset log for streaming query since
>>>>>>> the value of the config must be consistent during the lifecycle of the
>>>>>>> query. This means, the problematic config is already persisted for
>>>>>>> streaming query which ever ran with Spark 3.5.4.
>>>>>>>
>>>>>>> For the migration logic, we re-assign the value of the problematic
>>>>>>> config to the new config. This happens when the query is restarted, and it
>>>>>>> will be reflected into an offset log for "newer batch" so after a couple
>>>>>>> new microbatches the migration logic isn't needed. This migration logic is
>>>>>>> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
>>>>>>> couple microbatches, it will be mitigated.
>>>>>>>
>>>>>>> But I would say that there will always be a case that users just
>>>>>>> bump the minor/major version without following all the bugfix versions. I
>>>>>>> think it is still dangerous to remove the migration logic in Spark 4.0.0
>>>>>>> (and probably Spark 4.1.0, depending on the discussion). From the migration
>>>>>>> logic, the problematic config is just a "string", and users wouldn't be
>>>>>>> able to set the value with the problematic config name. We don't document
>>>>>>> this, as it'll be done automatically.
>>>>>>>
>>>>>>> That said, I'd propose to have migration logic for Spark 4.0 version
>>>>>>> line (at minimum, 4.1 is debatable). This will give a safer and less burden
>>>>>>> migration path for users with just retaining a problematic "string" (again,
>>>>>>> not a config).
>>>>>>>
>>>>>>> I'd love to hear the community's voice on this. I'd like to remind
>>>>>>> you, this is a blocker for Spark 4.0.0.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>
>>>>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Dongjoon Hyun <do...@gmail.com>]

Let me reformulate your suggestions and my interpretation.

Option 1 "Adding back `spark.databricks.*` in Spark codebase and keep
forever"

If we follow the proposed logic and reasoning, it means there is no safe
version to remove that configuration because Apache Spark 3.5.4 users can
jump to any future releases like Spark 4.1.0, 4.2.0, and 5.0.0 technically.
In other words, we cannot remove that logic forever.

That's the reason why we couldn't make an agreement so far.

Option 2 is simply adding a sentence (or more accurate one) for Spark 3.5.4
into the Spark 4.0.0 guideline because all other Spark versions (except
3.5.4) are not contaminated by `spark.databricks.*` conf.

"For Spark 3.5.4 streaming jobs, if you want to migrate the existing
running jobs, we need to upgrade them to Spark 3.5.5+ before upgrading
Spark 4.0"

Dongjoon.


On Tue, Mar 4, 2025 at 11:11 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Let's not start with VOTE right now, but let me make clear about options
> and pros/cons for the option, so that people can choose one over another.
>
> Option 1 (Current proposal): retain migration logic for Spark 4.0 (and
> maybe more minor versions, up to decision) which contains the problematic
> config as "string".
>
> Pros: We can avoid breaking users' queries in any path of upgrade, as long
> as we retain the migration logic. For example, we can support upgrading the
> streaming query which ever ran in Spark 3.5.4 to Spark 4.0.x as long as we
> decide to retain the migration logic for Spark 4.0. Spark 4.1.x, Spark
> 4.2.x, etc. as long as we retain the migration path longer.
> Cons: We retain the concerned config name in the codebase, though it's a
> string and users can never set it.
>
> Option 2 (Dongjoon's proposal): do not bring the migration logic in Spark
> 4.0 and force users to run existing streaming query with Spark 3.5.5+
> before upgrading to Spark 4.0.0+.
>
> Pros: We stop retaining the concerned config name in the codebase.
> Cons: Upgrading directly from Spark 3.5.4 to Spark 4.0+ will be missing
> the critical QO fix, which can lead to a "broken" checkpoint. If the
> checkpoint is broken, there is no way to restore and users have to restart
> the query from scratch. Since the target workload is stateful, in the worst
> case, the query has to start from the earliest data.
>
> I would only agree about the severity if the ASF project had a case of
> vendor name in codebase and it was decided to pay whatever cost to fix the
> case. I'm happy to be corrected if we have the doc in ASF explicitly
> mentioning the case and action item.
>
> On Wed, Mar 5, 2025 at 3:51 PM Wenchen Fan <cl...@gmail.com> wrote:
>
>> Shall we open an official vote for it? We can put more details on it so
>> that people can vote:
>> 1. how does it break user workloads without this migration code?
>> 2. what is the Apache policy for leaked vendor names in the codebase? I
>> think this is not the only one, we also mentioned
>> `com.databricks.spark.csv` in
>> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L621C8-L621C32
>>
>> On Wed, Mar 5, 2025 at 2:40 PM Jungtaek Lim <ka...@gmail.com>
>> wrote:
>>
>>> One major question: How do you believe that we can enforce users on
>>> upgrading path? I have seen a bunch of cases where users upgrade 2-3 minor
>>> versions at once. Do you really believe we can just break their query?
>>> What's the data backing up your claim?
>>>
>>> I think we agree to disagree. I really don't want "users" to get into
>>> situations just because of us. It's regardless of who made the mistake -
>>> it's about what's the proper mitigation for this, and I do not believe
>>> enforcing users to upgrade to Spark 3.5.8+ before upgrading Spark 4.0 is a
>>> valid approach.
>>>
>>> If I could vote for your alternative option, I'm -1 for it.
>>>
>>> On Wed, Mar 5, 2025 at 3:29 PM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> Technically, there is no agreement here. In other words, we have the
>>>> same situation with the initial discussion thread where we couldn't build a
>>>> community consensus on this.
>>>>
>>>> > I will consider this as "lazy consensus" if there are no objections
>>>> > for 3 days from initiation of the thread.
>>>>
>>>> If you need an explicit veto, here is mine, -1, because I don't think
>>>> that's just a string.
>>>>
>>>> > the problematic config is just a "string",
>>>>
>>>> To be clear, as I proposed both in the PR comments and initial
>>>> discussion thread, I believe we had better keep the AS-IS `master` and
>>>> `branch-4.0` and recommend to upgrade to the latest version of Apache Spark
>>>> 3.5.x first before upgrading to Spark 4.
>>>>
>>>> Sincerely,
>>>> Dongjoon.
>>>>
>>>>
>>>> On Tue, Mar 4, 2025 at 8:37 PM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>>
>>>>> Bumping on this. Again, this is a blocker for Spark 4.0.0. I will
>>>>> consider this as "lazy consensus" if there are no objections for 3 days
>>>>> from initiation of the thread.
>>>>>
>>>>> On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <
>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>
>>>>>> Hi dev,
>>>>>>
>>>>>> This is a spin-up of the original thread "Deprecating and banning
>>>>>> `spark.databricks.*` config from Apache Spark repository". (link
>>>>>> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>>>>>>
>>>>>> From the original thread, we decided to deprecate the config in Spark
>>>>>> 3.5.5 and remove the config in Spark 4.0.0. That thread did not decide one
>>>>>> thing, about smooth migration logic.
>>>>>>
>>>>>> We "persist" the config into offset log for streaming query since the
>>>>>> value of the config must be consistent during the lifecycle of the query.
>>>>>> This means, the problematic config is already persisted for streaming query
>>>>>> which ever ran with Spark 3.5.4.
>>>>>>
>>>>>> For the migration logic, we re-assign the value of the problematic
>>>>>> config to the new config. This happens when the query is restarted, and it
>>>>>> will be reflected into an offset log for "newer batch" so after a couple
>>>>>> new microbatches the migration logic isn't needed. This migration logic is
>>>>>> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
>>>>>> couple microbatches, it will be mitigated.
>>>>>>
>>>>>> But I would say that there will always be a case that users just bump
>>>>>> the minor/major version without following all the bugfix versions. I think
>>>>>> it is still dangerous to remove the migration logic in Spark 4.0.0 (and
>>>>>> probably Spark 4.1.0, depending on the discussion). From the migration
>>>>>> logic, the problematic config is just a "string", and users wouldn't be
>>>>>> able to set the value with the problematic config name. We don't document
>>>>>> this, as it'll be done automatically.
>>>>>>
>>>>>> That said, I'd propose to have migration logic for Spark 4.0 version
>>>>>> line (at minimum, 4.1 is debatable). This will give a safer and less burden
>>>>>> migration path for users with just retaining a problematic "string" (again,
>>>>>> not a config).
>>>>>>
>>>>>> I'd love to hear the community's voice on this. I'd like to remind
>>>>>> you, this is a blocker for Spark 4.0.0.
>>>>>>
>>>>>> Thanks,
>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>
>>>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Let's not start with VOTE right now, but let me make clear about options
and pros/cons for the option, so that people can choose one over another.

Option 1 (Current proposal): retain migration logic for Spark 4.0 (and
maybe more minor versions, up to decision) which contains the problematic
config as "string".

Pros: We can avoid breaking users' queries in any path of upgrade, as long
as we retain the migration logic. For example, we can support upgrading the
streaming query which ever ran in Spark 3.5.4 to Spark 4.0.x as long as we
decide to retain the migration logic for Spark 4.0. Spark 4.1.x, Spark
4.2.x, etc. as long as we retain the migration path longer.
Cons: We retain the concerned config name in the codebase, though it's a
string and users can never set it.

Option 2 (Dongjoon's proposal): do not bring the migration logic in Spark
4.0 and force users to run existing streaming query with Spark 3.5.5+
before upgrading to Spark 4.0.0+.

Pros: We stop retaining the concerned config name in the codebase.
Cons: Upgrading directly from Spark 3.5.4 to Spark 4.0+ will be missing the
critical QO fix, which can lead to a "broken" checkpoint. If the checkpoint
is broken, there is no way to restore and users have to restart the query
from scratch. Since the target workload is stateful, in the worst case, the
query has to start from the earliest data.

I would only agree about the severity if the ASF project had a case of
vendor name in codebase and it was decided to pay whatever cost to fix the
case. I'm happy to be corrected if we have the doc in ASF explicitly
mentioning the case and action item.

On Wed, Mar 5, 2025 at 3:51 PM Wenchen Fan <cl...@gmail.com> wrote:

> Shall we open an official vote for it? We can put more details on it so
> that people can vote:
> 1. how does it break user workloads without this migration code?
> 2. what is the Apache policy for leaked vendor names in the codebase? I
> think this is not the only one, we also mentioned
> `com.databricks.spark.csv` in
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L621C8-L621C32
>
> On Wed, Mar 5, 2025 at 2:40 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> One major question: How do you believe that we can enforce users on
>> upgrading path? I have seen a bunch of cases where users upgrade 2-3 minor
>> versions at once. Do you really believe we can just break their query?
>> What's the data backing up your claim?
>>
>> I think we agree to disagree. I really don't want "users" to get into
>> situations just because of us. It's regardless of who made the mistake -
>> it's about what's the proper mitigation for this, and I do not believe
>> enforcing users to upgrade to Spark 3.5.8+ before upgrading Spark 4.0 is a
>> valid approach.
>>
>> If I could vote for your alternative option, I'm -1 for it.
>>
>> On Wed, Mar 5, 2025 at 3:29 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Technically, there is no agreement here. In other words, we have the
>>> same situation with the initial discussion thread where we couldn't build a
>>> community consensus on this.
>>>
>>> > I will consider this as "lazy consensus" if there are no objections
>>> > for 3 days from initiation of the thread.
>>>
>>> If you need an explicit veto, here is mine, -1, because I don't think
>>> that's just a string.
>>>
>>> > the problematic config is just a "string",
>>>
>>> To be clear, as I proposed both in the PR comments and initial
>>> discussion thread, I believe we had better keep the AS-IS `master` and
>>> `branch-4.0` and recommend to upgrade to the latest version of Apache Spark
>>> 3.5.x first before upgrading to Spark 4.
>>>
>>> Sincerely,
>>> Dongjoon.
>>>
>>>
>>> On Tue, Mar 4, 2025 at 8:37 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> Bumping on this. Again, this is a blocker for Spark 4.0.0. I will
>>>> consider this as "lazy consensus" if there are no objections for 3 days
>>>> from initiation of the thread.
>>>>
>>>> On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>>
>>>>> Hi dev,
>>>>>
>>>>> This is a spin-up of the original thread "Deprecating and banning
>>>>> `spark.databricks.*` config from Apache Spark repository". (link
>>>>> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>>>>>
>>>>> From the original thread, we decided to deprecate the config in Spark
>>>>> 3.5.5 and remove the config in Spark 4.0.0. That thread did not decide one
>>>>> thing, about smooth migration logic.
>>>>>
>>>>> We "persist" the config into offset log for streaming query since the
>>>>> value of the config must be consistent during the lifecycle of the query.
>>>>> This means, the problematic config is already persisted for streaming query
>>>>> which ever ran with Spark 3.5.4.
>>>>>
>>>>> For the migration logic, we re-assign the value of the problematic
>>>>> config to the new config. This happens when the query is restarted, and it
>>>>> will be reflected into an offset log for "newer batch" so after a couple
>>>>> new microbatches the migration logic isn't needed. This migration logic is
>>>>> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
>>>>> couple microbatches, it will be mitigated.
>>>>>
>>>>> But I would say that there will always be a case that users just bump
>>>>> the minor/major version without following all the bugfix versions. I think
>>>>> it is still dangerous to remove the migration logic in Spark 4.0.0 (and
>>>>> probably Spark 4.1.0, depending on the discussion). From the migration
>>>>> logic, the problematic config is just a "string", and users wouldn't be
>>>>> able to set the value with the problematic config name. We don't document
>>>>> this, as it'll be done automatically.
>>>>>
>>>>> That said, I'd propose to have migration logic for Spark 4.0 version
>>>>> line (at minimum, 4.1 is debatable). This will give a safer and less burden
>>>>> migration path for users with just retaining a problematic "string" (again,
>>>>> not a config).
>>>>>
>>>>> I'd love to hear the community's voice on this. I'd like to remind
>>>>> you, this is a blocker for Spark 4.0.0.
>>>>>
>>>>> Thanks,
>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Wenchen Fan <cl...@gmail.com>]

Shall we open an official vote for it? We can put more details on it so
that people can vote:
1. how does it break user workloads without this migration code?
2. what is the Apache policy for leaked vendor names in the codebase? I
think this is not the only one, we also mentioned
`com.databricks.spark.csv` in
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L621C8-L621C32

On Wed, Mar 5, 2025 at 2:40 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> One major question: How do you believe that we can enforce users on
> upgrading path? I have seen a bunch of cases where users upgrade 2-3 minor
> versions at once. Do you really believe we can just break their query?
> What's the data backing up your claim?
>
> I think we agree to disagree. I really don't want "users" to get into
> situations just because of us. It's regardless of who made the mistake -
> it's about what's the proper mitigation for this, and I do not believe
> enforcing users to upgrade to Spark 3.5.8+ before upgrading Spark 4.0 is a
> valid approach.
>
> If I could vote for your alternative option, I'm -1 for it.
>
> On Wed, Mar 5, 2025 at 3:29 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Technically, there is no agreement here. In other words, we have the
>> same situation with the initial discussion thread where we couldn't build a
>> community consensus on this.
>>
>> > I will consider this as "lazy consensus" if there are no objections
>> > for 3 days from initiation of the thread.
>>
>> If you need an explicit veto, here is mine, -1, because I don't think
>> that's just a string.
>>
>> > the problematic config is just a "string",
>>
>> To be clear, as I proposed both in the PR comments and initial discussion
>> thread, I believe we had better keep the AS-IS `master` and `branch-4.0`
>> and recommend to upgrade to the latest version of Apache Spark 3.5.x first
>> before upgrading to Spark 4.
>>
>> Sincerely,
>> Dongjoon.
>>
>>
>> On Tue, Mar 4, 2025 at 8:37 PM Jungtaek Lim <ka...@gmail.com>
>> wrote:
>>
>>> Bumping on this. Again, this is a blocker for Spark 4.0.0. I will
>>> consider this as "lazy consensus" if there are no objections for 3 days
>>> from initiation of the thread.
>>>
>>> On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> Hi dev,
>>>>
>>>> This is a spin-up of the original thread "Deprecating and banning
>>>> `spark.databricks.*` config from Apache Spark repository". (link
>>>> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>>>>
>>>> From the original thread, we decided to deprecate the config in Spark
>>>> 3.5.5 and remove the config in Spark 4.0.0. That thread did not decide one
>>>> thing, about smooth migration logic.
>>>>
>>>> We "persist" the config into offset log for streaming query since the
>>>> value of the config must be consistent during the lifecycle of the query.
>>>> This means, the problematic config is already persisted for streaming query
>>>> which ever ran with Spark 3.5.4.
>>>>
>>>> For the migration logic, we re-assign the value of the problematic
>>>> config to the new config. This happens when the query is restarted, and it
>>>> will be reflected into an offset log for "newer batch" so after a couple
>>>> new microbatches the migration logic isn't needed. This migration logic is
>>>> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
>>>> couple microbatches, it will be mitigated.
>>>>
>>>> But I would say that there will always be a case that users just bump
>>>> the minor/major version without following all the bugfix versions. I think
>>>> it is still dangerous to remove the migration logic in Spark 4.0.0 (and
>>>> probably Spark 4.1.0, depending on the discussion). From the migration
>>>> logic, the problematic config is just a "string", and users wouldn't be
>>>> able to set the value with the problematic config name. We don't document
>>>> this, as it'll be done automatically.
>>>>
>>>> That said, I'd propose to have migration logic for Spark 4.0 version
>>>> line (at minimum, 4.1 is debatable). This will give a safer and less burden
>>>> migration path for users with just retaining a problematic "string" (again,
>>>> not a config).
>>>>
>>>> I'd love to hear the community's voice on this. I'd like to remind you,
>>>> this is a blocker for Spark 4.0.0.
>>>>
>>>> Thanks,
>>>> Jungtaek Lim (HeartSaVioR)
>>>>
>>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

One major question: How do you believe that we can enforce users on
upgrading path? I have seen a bunch of cases where users upgrade 2-3 minor
versions at once. Do you really believe we can just break their query?
What's the data backing up your claim?

I think we agree to disagree. I really don't want "users" to get into
situations just because of us. It's regardless of who made the mistake -
it's about what's the proper mitigation for this, and I do not believe
enforcing users to upgrade to Spark 3.5.8+ before upgrading Spark 4.0 is a
valid approach.

If I could vote for your alternative option, I'm -1 for it.

On Wed, Mar 5, 2025 at 3:29 PM Dongjoon Hyun <do...@gmail.com>
wrote:

> Technically, there is no agreement here. In other words, we have the same
> situation with the initial discussion thread where we couldn't build a
> community consensus on this.
>
> > I will consider this as "lazy consensus" if there are no objections
> > for 3 days from initiation of the thread.
>
> If you need an explicit veto, here is mine, -1, because I don't think
> that's just a string.
>
> > the problematic config is just a "string",
>
> To be clear, as I proposed both in the PR comments and initial discussion
> thread, I believe we had better keep the AS-IS `master` and `branch-4.0`
> and recommend to upgrade to the latest version of Apache Spark 3.5.x first
> before upgrading to Spark 4.
>
> Sincerely,
> Dongjoon.
>
>
> On Tue, Mar 4, 2025 at 8:37 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Bumping on this. Again, this is a blocker for Spark 4.0.0. I will
>> consider this as "lazy consensus" if there are no objections for 3 days
>> from initiation of the thread.
>>
>> On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <ka...@gmail.com>
>> wrote:
>>
>>> Hi dev,
>>>
>>> This is a spin-up of the original thread "Deprecating and banning
>>> `spark.databricks.*` config from Apache Spark repository". (link
>>> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>>>
>>> From the original thread, we decided to deprecate the config in Spark
>>> 3.5.5 and remove the config in Spark 4.0.0. That thread did not decide one
>>> thing, about smooth migration logic.
>>>
>>> We "persist" the config into offset log for streaming query since the
>>> value of the config must be consistent during the lifecycle of the query.
>>> This means, the problematic config is already persisted for streaming query
>>> which ever ran with Spark 3.5.4.
>>>
>>> For the migration logic, we re-assign the value of the problematic
>>> config to the new config. This happens when the query is restarted, and it
>>> will be reflected into an offset log for "newer batch" so after a couple
>>> new microbatches the migration logic isn't needed. This migration logic is
>>> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
>>> couple microbatches, it will be mitigated.
>>>
>>> But I would say that there will always be a case that users just bump
>>> the minor/major version without following all the bugfix versions. I think
>>> it is still dangerous to remove the migration logic in Spark 4.0.0 (and
>>> probably Spark 4.1.0, depending on the discussion). From the migration
>>> logic, the problematic config is just a "string", and users wouldn't be
>>> able to set the value with the problematic config name. We don't document
>>> this, as it'll be done automatically.
>>>
>>> That said, I'd propose to have migration logic for Spark 4.0 version
>>> line (at minimum, 4.1 is debatable). This will give a safer and less burden
>>> migration path for users with just retaining a problematic "string" (again,
>>> not a config).
>>>
>>> I'd love to hear the community's voice on this. I'd like to remind you,
>>> this is a blocker for Spark 4.0.0.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Dongjoon Hyun <do...@gmail.com>]

Technically, there is no agreement here. In other words, we have the same
situation with the initial discussion thread where we couldn't build a
community consensus on this.

> I will consider this as "lazy consensus" if there are no objections
> for 3 days from initiation of the thread.

If you need an explicit veto, here is mine, -1, because I don't think
that's just a string.

> the problematic config is just a "string",

To be clear, as I proposed both in the PR comments and initial discussion
thread, I believe we had better keep the AS-IS `master` and `branch-4.0`
and recommend to upgrade to the latest version of Apache Spark 3.5.x first
before upgrading to Spark 4.

Sincerely,
Dongjoon.


On Tue, Mar 4, 2025 at 8:37 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Bumping on this. Again, this is a blocker for Spark 4.0.0. I will consider
> this as "lazy consensus" if there are no objections for 3 days from
> initiation of the thread.
>
> On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Hi dev,
>>
>> This is a spin-up of the original thread "Deprecating and banning
>> `spark.databricks.*` config from Apache Spark repository". (link
>> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>>
>> From the original thread, we decided to deprecate the config in Spark
>> 3.5.5 and remove the config in Spark 4.0.0. That thread did not decide one
>> thing, about smooth migration logic.
>>
>> We "persist" the config into offset log for streaming query since the
>> value of the config must be consistent during the lifecycle of the query.
>> This means, the problematic config is already persisted for streaming query
>> which ever ran with Spark 3.5.4.
>>
>> For the migration logic, we re-assign the value of the problematic config
>> to the new config. This happens when the query is restarted, and it will be
>> reflected into an offset log for "newer batch" so after a couple new
>> microbatches the migration logic isn't needed. This migration logic is
>> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
>> couple microbatches, it will be mitigated.
>>
>> But I would say that there will always be a case that users just bump the
>> minor/major version without following all the bugfix versions. I think it
>> is still dangerous to remove the migration logic in Spark 4.0.0 (and
>> probably Spark 4.1.0, depending on the discussion). From the migration
>> logic, the problematic config is just a "string", and users wouldn't be
>> able to set the value with the problematic config name. We don't document
>> this, as it'll be done automatically.
>>
>> That said, I'd propose to have migration logic for Spark 4.0 version line
>> (at minimum, 4.1 is debatable). This will give a safer and less burden
>> migration path for users with just retaining a problematic "string" (again,
>> not a config).
>>
>> I'd love to hear the community's voice on this. I'd like to remind you,
>> this is a blocker for Spark 4.0.0.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

[Jungtaek Lim <ka...@gmail.com>]

Bumping on this. Again, this is a blocker for Spark 4.0.0. I will consider
this as "lazy consensus" if there are no objections for 3 days from
initiation of the thread.

On Tue, Mar 4, 2025 at 2:15 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Hi dev,
>
> This is a spin-up of the original thread "Deprecating and banning
> `spark.databricks.*` config from Apache Spark repository". (link
> <https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd>)
>
> From the original thread, we decided to deprecate the config in Spark
> 3.5.5 and remove the config in Spark 4.0.0. That thread did not decide one
> thing, about smooth migration logic.
>
> We "persist" the config into offset log for streaming query since the
> value of the config must be consistent during the lifecycle of the query.
> This means, the problematic config is already persisted for streaming query
> which ever ran with Spark 3.5.4.
>
> For the migration logic, we re-assign the value of the problematic config
> to the new config. This happens when the query is restarted, and it will be
> reflected into an offset log for "newer batch" so after a couple new
> microbatches the migration logic isn't needed. This migration logic is
> shipped in Spark 3.5.5, so once the query is run with Spark 3.5.5 for a
> couple microbatches, it will be mitigated.
>
> But I would say that there will always be a case that users just bump the
> minor/major version without following all the bugfix versions. I think it
> is still dangerous to remove the migration logic in Spark 4.0.0 (and
> probably Spark 4.1.0, depending on the discussion). From the migration
> logic, the problematic config is just a "string", and users wouldn't be
> able to set the value with the problematic config name. We don't document
> this, as it'll be done automatically.
>
> That said, I'd propose to have migration logic for Spark 4.0 version line
> (at minimum, 4.1 is debatable). This will give a safer and less burden
> migration path for users with just retaining a problematic "string" (again,
> not a config).
>
> I'd love to hear the community's voice on this. I'd like to remind you,
> this is a blocker for Spark 4.0.0.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>