[VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Mark Hamstra <ma...@gmail.com>]

There has been enough discussion on this topic already, so I think
that an immediate vote on the validity of Dongjoon's technical
justification for his veto of the "Retain migration logic ... in Spark
4.0.x" proposal is in order. That technical justification has been
called into question, and the guidance at
https://www.apache.org/foundation/glossary.html#Veto leaves it to the
PMC to determine whether the technical justification is  valid: "In
case of doubt, deciding whether a technical justification is valid is
up to the PMC." As such, only PMC votes will decide the outcome of
this vote. This is neither a vote on a code change itself not a vote
on whether a package is ready for release, so it a procedural vote on
whether the technical justification is valid. As such, the vote will
be decided by a simple majority where +1 votes hold that the technical
justification is not valid and -1 votes hold that the technical
justification is valid.

I would request that at least PMC members post more than just a naked
vote, but instead endeavor to give some reason why they have assessed
the technical justification as they have. I'll start:

Despite all of the discussion related to Dongjoon's -1 vote, I must
confess to still not being entirely clear on what is his technical
justification for that veto. I see claims that including an admonition
in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
required to maintain the integrity of already existing data streams,
and I see assertions about the maintenance burden that including the
migration logic would impose on future Spark versions, but I don't
think that I see any other technical objections. I do not believe that
the claimed technical justification is valid.

In requiring that a veto of a code change be accompanied by a
technical justification for the veto, the Apache Voting Process states
that: "To prevent vetoes from being used capriciously, the voter must
provide with the veto a technical justification showing why the change
is bad (opens a security exposure, negatively affects performance,
etc. ). A veto without a justification is invalid and has no weight."
This strongly implies that there must be something objectively wrong
with the proposed code change in that it causes significant harm in
the way of opening a security exposure, negatively affecting
performance, or presumably other significant user harms or perhaps
even developer burdens.

The proposed addition of the migration logic to Spark 4.0.x does not
cause any harm to Spark's users. For many users, those not using
streaming data, the change will have no effect. For streaming users
the change will be beneficial, not harmful.

Neither do I find the claim of excessive, ongoing developer burden to
be persuasive. The changes are tiny and easily maintained -- in fact,
it wouldn't surprise me if no further changes to this migration logic
would be needed for a very long time.

Some of what we are left with is just an expression of preference for
a technical alternative to the migration logic -- i.e. including in
the release notes an admonition to first upgrade to 3.5.5. But the
Apache Voting Process does not say that in the face of code
alternatives A and B, a qualified voter is justified in vetoing A if
they prefer B. Instead, the Voting Process strongly implies that
something more is needed to justify a veto, as I've already covered.
Thus I don't find Dongjoon's preference for the release notes option
to be adequate justification for the veto.

The only remaining question I see is whether including "databricks" in
the Apache Code is ever allowed or if any such instance must be
expunged as soon as possible. I am not aware of any ASF policy that
strictly forbids the mention of a vendor in Apache code for any
reason, even if that vendor has a product based on Apache code, even
if that vendor enjoys a uniquely influential position vis a vis some
Apache code or project. Certainly the PMC has a duty to see to it that
neither Databricks nor any other vendor exercises influence or control
over Apache Spark outside of the established Apache process, but the
proposed migration code changes do not advantage Databricks -- if
anything they remove a minor avenue of influence, and simply need to
mention "databricks" once in order match and transform a configuration
into a vendor neutral equivalent. While not optimal, I can't find such
a one-time inclusion of "databricks" to be truly offensive to any
non-technical policy concern -- certainly not offensive to the point
that it outweighs the user advantage of including the migration logic
in Spark 4.0.x.

In summary, I do not find Dongjoon's given technical justification to
be valid relative to the Apache requirements for a veto of a code
change, so I must vote...

+1

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Reynold Xin <rx...@databricks.com.INVALID>]

Thanks Mark for starting this. +1 and agree with your reasoning.

Wearing the Apache Spark PMC hat, I think having a few lines of
straightforward logic to ease users' migrations is a no-brainer to do.
Imagine how confused a user would be when they upgraded to 4.0 and things
stopped working in a way that's not obvious ...

Wearing the Databricks hat, I honestly don't care about this issue at all.
Our customers will have no trouble upgrading regardless of this change.



On Sat, Mar 15, 2025 at 4:38 PM Mark Hamstra <ma...@gmail.com> wrote:

> There has been enough discussion on this topic already, so I think
> that an immediate vote on the validity of Dongjoon's technical
> justification for his veto of the "Retain migration logic ... in Spark
> 4.0.x" proposal is in order. That technical justification has been
> called into question, and the guidance at
> https://www.apache.org/foundation/glossary.html#Veto leaves it to the
> PMC to determine whether the technical justification is  valid: "In
> case of doubt, deciding whether a technical justification is valid is
> up to the PMC." As such, only PMC votes will decide the outcome of
> this vote. This is neither a vote on a code change itself not a vote
> on whether a package is ready for release, so it a procedural vote on
> whether the technical justification is valid. As such, the vote will
> be decided by a simple majority where +1 votes hold that the technical
> justification is not valid and -1 votes hold that the technical
> justification is valid.
>
> I would request that at least PMC members post more than just a naked
> vote, but instead endeavor to give some reason why they have assessed
> the technical justification as they have. I'll start:
>
> Despite all of the discussion related to Dongjoon's -1 vote, I must
> confess to still not being entirely clear on what is his technical
> justification for that veto. I see claims that including an admonition
> in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
> required to maintain the integrity of already existing data streams,
> and I see assertions about the maintenance burden that including the
> migration logic would impose on future Spark versions, but I don't
> think that I see any other technical objections. I do not believe that
> the claimed technical justification is valid.
>
> In requiring that a veto of a code change be accompanied by a
> technical justification for the veto, the Apache Voting Process states
> that: "To prevent vetoes from being used capriciously, the voter must
> provide with the veto a technical justification showing why the change
> is bad (opens a security exposure, negatively affects performance,
> etc. ). A veto without a justification is invalid and has no weight."
> This strongly implies that there must be something objectively wrong
> with the proposed code change in that it causes significant harm in
> the way of opening a security exposure, negatively affecting
> performance, or presumably other significant user harms or perhaps
> even developer burdens.
>
> The proposed addition of the migration logic to Spark 4.0.x does not
> cause any harm to Spark's users. For many users, those not using
> streaming data, the change will have no effect. For streaming users
> the change will be beneficial, not harmful.
>
> Neither do I find the claim of excessive, ongoing developer burden to
> be persuasive. The changes are tiny and easily maintained -- in fact,
> it wouldn't surprise me if no further changes to this migration logic
> would be needed for a very long time.
>
> Some of what we are left with is just an expression of preference for
> a technical alternative to the migration logic -- i.e. including in
> the release notes an admonition to first upgrade to 3.5.5. But the
> Apache Voting Process does not say that in the face of code
> alternatives A and B, a qualified voter is justified in vetoing A if
> they prefer B. Instead, the Voting Process strongly implies that
> something more is needed to justify a veto, as I've already covered.
> Thus I don't find Dongjoon's preference for the release notes option
> to be adequate justification for the veto.
>
> The only remaining question I see is whether including "databricks" in
> the Apache Code is ever allowed or if any such instance must be
> expunged as soon as possible. I am not aware of any ASF policy that
> strictly forbids the mention of a vendor in Apache code for any
> reason, even if that vendor has a product based on Apache code, even
> if that vendor enjoys a uniquely influential position vis a vis some
> Apache code or project. Certainly the PMC has a duty to see to it that
> neither Databricks nor any other vendor exercises influence or control
> over Apache Spark outside of the established Apache process, but the
> proposed migration code changes do not advantage Databricks -- if
> anything they remove a minor avenue of influence, and simply need to
> mention "databricks" once in order match and transform a configuration
> into a vendor neutral equivalent. While not optimal, I can't find such
> a one-time inclusion of "databricks" to be truly offensive to any
> non-technical policy concern -- certainly not offensive to the point
> that it outweighs the user advantage of including the migration logic
> in Spark 4.0.x.
>
> In summary, I do not find Dongjoon's given technical justification to
> be valid relative to the Apache requirements for a veto of a code
> change, so I must vote...
>
> +1
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Wenchen Fan <cl...@gmail.com>]

Hi Holden,

> I think I disagree with Mark on the assertion that the veto needs to have
“substantial technical concern,” but rather a valid concern. I think in
addition to the veto they’ve also gone above and beyond providing
alternative ways to accomplish this.

I think this is not what the Apache policy says in
https://www.apache.org/foundation/glossary.html#Veto : "*All vetoes must be
accompanied by a valid technical justification; a veto without such a
justification is invalid.*"

I think it's better for the Apache Spark community to follow the general
Apache policy.

On Mon, Mar 17, 2025 at 8:49 AM Wenchen Fan <cl...@gmail.com> wrote:

> Before I cast my vote here, I'd like to highlight one thing: As the
> release manager of Apache Spark 4.0.0, I was not notified about the
> breaking change of renaming an already-released configuration:
> https://github.com/apache/spark/pull/49897 . Note that the previous VOTE
> from Dongjoon was about Apache Spark 3.5.5, which means there is no
> consensus about what we should do for 4.0.0 yet. I think it's fair for me
> to ask to revert the breaking change and unblock 4.0.0, which is a common
> practice of how we handle breaking changes in Apache Spark and I don't
> think I need a VOTE for it.
>
> Of course, none of us want to keep the misnamed configuration in 4.0.0,
> and it’s clear to me that applying the “configuration deprecation” approach
> from 3.5.5 to 4.0.0 is the best path forward. I don’t believe Dongjoon’s
> veto has valid technical justification, so I’m +1 on this vote.
>
> Thanks,
> Wenchen
>
>
> On Mon, Mar 17, 2025 at 7:27 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> I was trying hard to stay away from this VOTE, but I should have reminded
>> everyone about "what" we are going to VOTE.
>>
>> Dongjoon casted a VETO against code change VOTE. That VETO is described
>> in ASF Voting Process page:
>>
>> https://www.apache.org/foundation/voting.html#Veto
>>
>> A -1 vote by a qualified voter stops a code-modification proposal in its
>>> tracks. This constitutes a veto, and it cannot be overruled nor overridden
>>> by anyone. Vetoes stand until and unless the individual withdraws their
>>> veto.
>>>
>>> To prevent vetoes from being used capriciously, the voter must provide
>>> with the veto a technical justification showing why the change is bad
>>> (opens a security exposure, negatively affects performance, etc. ). A veto
>>> without a justification is invalid and has no weight.
>>
>>
>> For sure, the technical justification must be "objective", otherwise it
>> means if I'm a PMC member I can veto everything if I don't like it.
>>
>> The main argument here about "vendor name in the codebase" is NOT
>> something we have ever seen disallowing this in ASF policy. If there is
>> evidence, it will immediately kill the two VOTEs as it is enough objective
>> argument. But no one was able to bring this up. Please remember, the fact
>> "vendor name in the codebase is bad for any reason", is proven to be NOT an
>> "objective" claim, otherwise how the DISCUSS and VOTE were almost passing
>> with support from PMC members?
>>
>> I really suggest everyone who casts a vote in this VOTE thread, to be
>> based on "objective" rationale. For example, we tend to consider < 10 lines
>> of code to be very trivial to maintain, so the argument of maintenance
>> burden does not apply here. Like this.
>>
>>
>> On Sun, Mar 16, 2025 at 8:38 AM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> There has been enough discussion on this topic already, so I think
>>> that an immediate vote on the validity of Dongjoon's technical
>>> justification for his veto of the "Retain migration logic ... in Spark
>>> 4.0.x" proposal is in order. That technical justification has been
>>> called into question, and the guidance at
>>> https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>>> PMC to determine whether the technical justification is  valid: "In
>>> case of doubt, deciding whether a technical justification is valid is
>>> up to the PMC." As such, only PMC votes will decide the outcome of
>>> this vote. This is neither a vote on a code change itself not a vote
>>> on whether a package is ready for release, so it a procedural vote on
>>> whether the technical justification is valid. As such, the vote will
>>> be decided by a simple majority where +1 votes hold that the technical
>>> justification is not valid and -1 votes hold that the technical
>>> justification is valid.
>>>
>>> I would request that at least PMC members post more than just a naked
>>> vote, but instead endeavor to give some reason why they have assessed
>>> the technical justification as they have. I'll start:
>>>
>>> Despite all of the discussion related to Dongjoon's -1 vote, I must
>>> confess to still not being entirely clear on what is his technical
>>> justification for that veto. I see claims that including an admonition
>>> in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>> required to maintain the integrity of already existing data streams,
>>> and I see assertions about the maintenance burden that including the
>>> migration logic would impose on future Spark versions, but I don't
>>> think that I see any other technical objections. I do not believe that
>>> the claimed technical justification is valid.
>>>
>>> In requiring that a veto of a code change be accompanied by a
>>> technical justification for the veto, the Apache Voting Process states
>>> that: "To prevent vetoes from being used capriciously, the voter must
>>> provide with the veto a technical justification showing why the change
>>> is bad (opens a security exposure, negatively affects performance,
>>> etc. ). A veto without a justification is invalid and has no weight."
>>> This strongly implies that there must be something objectively wrong
>>> with the proposed code change in that it causes significant harm in
>>> the way of opening a security exposure, negatively affecting
>>> performance, or presumably other significant user harms or perhaps
>>> even developer burdens.
>>>
>>> The proposed addition of the migration logic to Spark 4.0.x does not
>>> cause any harm to Spark's users. For many users, those not using
>>> streaming data, the change will have no effect. For streaming users
>>> the change will be beneficial, not harmful.
>>>
>>> Neither do I find the claim of excessive, ongoing developer burden to
>>> be persuasive. The changes are tiny and easily maintained -- in fact,
>>> it wouldn't surprise me if no further changes to this migration logic
>>> would be needed for a very long time.
>>>
>>> Some of what we are left with is just an expression of preference for
>>> a technical alternative to the migration logic -- i.e. including in
>>> the release notes an admonition to first upgrade to 3.5.5. But the
>>> Apache Voting Process does not say that in the face of code
>>> alternatives A and B, a qualified voter is justified in vetoing A if
>>> they prefer B. Instead, the Voting Process strongly implies that
>>> something more is needed to justify a veto, as I've already covered.
>>> Thus I don't find Dongjoon's preference for the release notes option
>>> to be adequate justification for the veto.
>>>
>>> The only remaining question I see is whether including "databricks" in
>>> the Apache Code is ever allowed or if any such instance must be
>>> expunged as soon as possible. I am not aware of any ASF policy that
>>> strictly forbids the mention of a vendor in Apache code for any
>>> reason, even if that vendor has a product based on Apache code, even
>>> if that vendor enjoys a uniquely influential position vis a vis some
>>> Apache code or project. Certainly the PMC has a duty to see to it that
>>> neither Databricks nor any other vendor exercises influence or control
>>> over Apache Spark outside of the established Apache process, but the
>>> proposed migration code changes do not advantage Databricks -- if
>>> anything they remove a minor avenue of influence, and simply need to
>>> mention "databricks" once in order match and transform a configuration
>>> into a vendor neutral equivalent. While not optimal, I can't find such
>>> a one-time inclusion of "databricks" to be truly offensive to any
>>> non-technical policy concern -- certainly not offensive to the point
>>> that it outweighs the user advantage of including the migration logic
>>> in Spark 4.0.x.
>>>
>>> In summary, I do not find Dongjoon's given technical justification to
>>> be valid relative to the Apache requirements for a veto of a code
>>> change, so I must vote...
>>>
>>> +1
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Wenchen Fan <cl...@gmail.com>]

Before I cast my vote here, I'd like to highlight one thing: As the release
manager of Apache Spark 4.0.0, I was not notified about the breaking change
of renaming an already-released configuration:
https://github.com/apache/spark/pull/49897 . Note that the previous VOTE
from Dongjoon was about Apache Spark 3.5.5, which means there is no
consensus about what we should do for 4.0.0 yet. I think it's fair for me
to ask to revert the breaking change and unblock 4.0.0, which is a common
practice of how we handle breaking changes in Apache Spark and I don't
think I need a VOTE for it.

Of course, none of us want to keep the misnamed configuration in 4.0.0, and
it’s clear to me that applying the “configuration deprecation” approach
from 3.5.5 to 4.0.0 is the best path forward. I don’t believe Dongjoon’s
veto has valid technical justification, so I’m +1 on this vote.

Thanks,
Wenchen


On Mon, Mar 17, 2025 at 7:27 AM Jungtaek Lim <ka...@gmail.com>
wrote:

> I was trying hard to stay away from this VOTE, but I should have reminded
> everyone about "what" we are going to VOTE.
>
> Dongjoon casted a VETO against code change VOTE. That VETO is described in
> ASF Voting Process page:
>
> https://www.apache.org/foundation/voting.html#Veto
>
> A -1 vote by a qualified voter stops a code-modification proposal in its
>> tracks. This constitutes a veto, and it cannot be overruled nor overridden
>> by anyone. Vetoes stand until and unless the individual withdraws their
>> veto.
>>
>> To prevent vetoes from being used capriciously, the voter must provide
>> with the veto a technical justification showing why the change is bad
>> (opens a security exposure, negatively affects performance, etc. ). A veto
>> without a justification is invalid and has no weight.
>
>
> For sure, the technical justification must be "objective", otherwise it
> means if I'm a PMC member I can veto everything if I don't like it.
>
> The main argument here about "vendor name in the codebase" is NOT
> something we have ever seen disallowing this in ASF policy. If there is
> evidence, it will immediately kill the two VOTEs as it is enough objective
> argument. But no one was able to bring this up. Please remember, the fact
> "vendor name in the codebase is bad for any reason", is proven to be NOT an
> "objective" claim, otherwise how the DISCUSS and VOTE were almost passing
> with support from PMC members?
>
> I really suggest everyone who casts a vote in this VOTE thread, to be
> based on "objective" rationale. For example, we tend to consider < 10 lines
> of code to be very trivial to maintain, so the argument of maintenance
> burden does not apply here. Like this.
>
>
> On Sun, Mar 16, 2025 at 8:38 AM Mark Hamstra <ma...@gmail.com>
> wrote:
>
>> There has been enough discussion on this topic already, so I think
>> that an immediate vote on the validity of Dongjoon's technical
>> justification for his veto of the "Retain migration logic ... in Spark
>> 4.0.x" proposal is in order. That technical justification has been
>> called into question, and the guidance at
>> https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>> PMC to determine whether the technical justification is  valid: "In
>> case of doubt, deciding whether a technical justification is valid is
>> up to the PMC." As such, only PMC votes will decide the outcome of
>> this vote. This is neither a vote on a code change itself not a vote
>> on whether a package is ready for release, so it a procedural vote on
>> whether the technical justification is valid. As such, the vote will
>> be decided by a simple majority where +1 votes hold that the technical
>> justification is not valid and -1 votes hold that the technical
>> justification is valid.
>>
>> I would request that at least PMC members post more than just a naked
>> vote, but instead endeavor to give some reason why they have assessed
>> the technical justification as they have. I'll start:
>>
>> Despite all of the discussion related to Dongjoon's -1 vote, I must
>> confess to still not being entirely clear on what is his technical
>> justification for that veto. I see claims that including an admonition
>> in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>> required to maintain the integrity of already existing data streams,
>> and I see assertions about the maintenance burden that including the
>> migration logic would impose on future Spark versions, but I don't
>> think that I see any other technical objections. I do not believe that
>> the claimed technical justification is valid.
>>
>> In requiring that a veto of a code change be accompanied by a
>> technical justification for the veto, the Apache Voting Process states
>> that: "To prevent vetoes from being used capriciously, the voter must
>> provide with the veto a technical justification showing why the change
>> is bad (opens a security exposure, negatively affects performance,
>> etc. ). A veto without a justification is invalid and has no weight."
>> This strongly implies that there must be something objectively wrong
>> with the proposed code change in that it causes significant harm in
>> the way of opening a security exposure, negatively affecting
>> performance, or presumably other significant user harms or perhaps
>> even developer burdens.
>>
>> The proposed addition of the migration logic to Spark 4.0.x does not
>> cause any harm to Spark's users. For many users, those not using
>> streaming data, the change will have no effect. For streaming users
>> the change will be beneficial, not harmful.
>>
>> Neither do I find the claim of excessive, ongoing developer burden to
>> be persuasive. The changes are tiny and easily maintained -- in fact,
>> it wouldn't surprise me if no further changes to this migration logic
>> would be needed for a very long time.
>>
>> Some of what we are left with is just an expression of preference for
>> a technical alternative to the migration logic -- i.e. including in
>> the release notes an admonition to first upgrade to 3.5.5. But the
>> Apache Voting Process does not say that in the face of code
>> alternatives A and B, a qualified voter is justified in vetoing A if
>> they prefer B. Instead, the Voting Process strongly implies that
>> something more is needed to justify a veto, as I've already covered.
>> Thus I don't find Dongjoon's preference for the release notes option
>> to be adequate justification for the veto.
>>
>> The only remaining question I see is whether including "databricks" in
>> the Apache Code is ever allowed or if any such instance must be
>> expunged as soon as possible. I am not aware of any ASF policy that
>> strictly forbids the mention of a vendor in Apache code for any
>> reason, even if that vendor has a product based on Apache code, even
>> if that vendor enjoys a uniquely influential position vis a vis some
>> Apache code or project. Certainly the PMC has a duty to see to it that
>> neither Databricks nor any other vendor exercises influence or control
>> over Apache Spark outside of the established Apache process, but the
>> proposed migration code changes do not advantage Databricks -- if
>> anything they remove a minor avenue of influence, and simply need to
>> mention "databricks" once in order match and transform a configuration
>> into a vendor neutral equivalent. While not optimal, I can't find such
>> a one-time inclusion of "databricks" to be truly offensive to any
>> non-technical policy concern -- certainly not offensive to the point
>> that it outweighs the user advantage of including the migration logic
>> in Spark 4.0.x.
>>
>> In summary, I do not find Dongjoon's given technical justification to
>> be valid relative to the Apache requirements for a veto of a code
>> change, so I must vote...
>>
>> +1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

I was trying hard to stay away from this VOTE, but I should have reminded
everyone about "what" we are going to VOTE.

Dongjoon casted a VETO against code change VOTE. That VETO is described in
ASF Voting Process page:

https://www.apache.org/foundation/voting.html#Veto

A -1 vote by a qualified voter stops a code-modification proposal in its
> tracks. This constitutes a veto, and it cannot be overruled nor overridden
> by anyone. Vetoes stand until and unless the individual withdraws their
> veto.
>
> To prevent vetoes from being used capriciously, the voter must provide
> with the veto a technical justification showing why the change is bad
> (opens a security exposure, negatively affects performance, etc. ). A veto
> without a justification is invalid and has no weight.


For sure, the technical justification must be "objective", otherwise it
means if I'm a PMC member I can veto everything if I don't like it.

The main argument here about "vendor name in the codebase" is NOT something
we have ever seen disallowing this in ASF policy. If there is evidence, it
will immediately kill the two VOTEs as it is enough objective argument. But
no one was able to bring this up. Please remember, the fact "vendor name in
the codebase is bad for any reason", is proven to be NOT an "objective"
claim, otherwise how the DISCUSS and VOTE were almost passing with support
from PMC members?

I really suggest everyone who casts a vote in this VOTE thread, to be based
on "objective" rationale. For example, we tend to consider < 10 lines of
code to be very trivial to maintain, so the argument of maintenance burden
does not apply here. Like this.


On Sun, Mar 16, 2025 at 8:38 AM Mark Hamstra <ma...@gmail.com> wrote:

> There has been enough discussion on this topic already, so I think
> that an immediate vote on the validity of Dongjoon's technical
> justification for his veto of the "Retain migration logic ... in Spark
> 4.0.x" proposal is in order. That technical justification has been
> called into question, and the guidance at
> https://www.apache.org/foundation/glossary.html#Veto leaves it to the
> PMC to determine whether the technical justification is  valid: "In
> case of doubt, deciding whether a technical justification is valid is
> up to the PMC." As such, only PMC votes will decide the outcome of
> this vote. This is neither a vote on a code change itself not a vote
> on whether a package is ready for release, so it a procedural vote on
> whether the technical justification is valid. As such, the vote will
> be decided by a simple majority where +1 votes hold that the technical
> justification is not valid and -1 votes hold that the technical
> justification is valid.
>
> I would request that at least PMC members post more than just a naked
> vote, but instead endeavor to give some reason why they have assessed
> the technical justification as they have. I'll start:
>
> Despite all of the discussion related to Dongjoon's -1 vote, I must
> confess to still not being entirely clear on what is his technical
> justification for that veto. I see claims that including an admonition
> in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
> required to maintain the integrity of already existing data streams,
> and I see assertions about the maintenance burden that including the
> migration logic would impose on future Spark versions, but I don't
> think that I see any other technical objections. I do not believe that
> the claimed technical justification is valid.
>
> In requiring that a veto of a code change be accompanied by a
> technical justification for the veto, the Apache Voting Process states
> that: "To prevent vetoes from being used capriciously, the voter must
> provide with the veto a technical justification showing why the change
> is bad (opens a security exposure, negatively affects performance,
> etc. ). A veto without a justification is invalid and has no weight."
> This strongly implies that there must be something objectively wrong
> with the proposed code change in that it causes significant harm in
> the way of opening a security exposure, negatively affecting
> performance, or presumably other significant user harms or perhaps
> even developer burdens.
>
> The proposed addition of the migration logic to Spark 4.0.x does not
> cause any harm to Spark's users. For many users, those not using
> streaming data, the change will have no effect. For streaming users
> the change will be beneficial, not harmful.
>
> Neither do I find the claim of excessive, ongoing developer burden to
> be persuasive. The changes are tiny and easily maintained -- in fact,
> it wouldn't surprise me if no further changes to this migration logic
> would be needed for a very long time.
>
> Some of what we are left with is just an expression of preference for
> a technical alternative to the migration logic -- i.e. including in
> the release notes an admonition to first upgrade to 3.5.5. But the
> Apache Voting Process does not say that in the face of code
> alternatives A and B, a qualified voter is justified in vetoing A if
> they prefer B. Instead, the Voting Process strongly implies that
> something more is needed to justify a veto, as I've already covered.
> Thus I don't find Dongjoon's preference for the release notes option
> to be adequate justification for the veto.
>
> The only remaining question I see is whether including "databricks" in
> the Apache Code is ever allowed or if any such instance must be
> expunged as soon as possible. I am not aware of any ASF policy that
> strictly forbids the mention of a vendor in Apache code for any
> reason, even if that vendor has a product based on Apache code, even
> if that vendor enjoys a uniquely influential position vis a vis some
> Apache code or project. Certainly the PMC has a duty to see to it that
> neither Databricks nor any other vendor exercises influence or control
> over Apache Spark outside of the established Apache process, but the
> proposed migration code changes do not advantage Databricks -- if
> anything they remove a minor avenue of influence, and simply need to
> mention "databricks" once in order match and transform a configuration
> into a vendor neutral equivalent. While not optimal, I can't find such
> a one-time inclusion of "databricks" to be truly offensive to any
> non-technical policy concern -- certainly not offensive to the point
> that it outweighs the user advantage of including the migration logic
> in Spark 4.0.x.
>
> In summary, I do not find Dongjoon's given technical justification to
> be valid relative to the Apache requirements for a veto of a code
> change, so I must vote...
>
> +1
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Mich Talebzadeh <mi...@gmail.com>]

Ok guys great discussion all around.

it would be* interesting* to observe sentiments by participants in this
thread. The source is from spark dev archive for this topic as attached.

The code uses Python with the relevant libraries see the code and imbedded
explanation

*Approach for Sentiment Analysis*

*Data Extraction using Python*
Parsed the text file to extract messages per user from the attached file
from spark dev archive
Processed each message individually.
*Sentiment Scoring*
Used NLTK's VADER (Valence Aware Dictionary and sEntiment Reasoner)
<https://www.sciencebuddies.org/science-fair-projects/project-ideas/ArtificialIntelligence_p023/artificial-intelligence/sentiment_analysis#:~:text=Sentiment%20analysis%20is%20a%20technique%20in%20Natural%20Language,the%20overall%20mood%20is%20positive%2C%20negative%2C%20or%20neutral.>
to analyze the sentiment with help from AI
Assigned each message a score based on positive, neutral, and negative
sentiment expressed
*Aggregation:*
Computed an average sentiment score per user by taking the mean sentiment
across all their messages.

The final sentiment score for each user was calculated as:

Average Sentiment Score = (Sum of Compound Sentiment Scores) / (Total
Messages Sent)

[image: sentiment_score.png]
Dongjoon sentiment seems to be pretty neutral and the rest mildly positive

HTH



Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR

   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>





On Sun, 16 Mar 2025 at 02:22, Jungtaek Lim <ka...@gmail.com>
wrote:

> For sure, I’m +1 (non-binding).
>
> I believe I don’t need to explain more and I spent whole weekend and we
> have a history about my justification in the history of mailing list.
>
> I’m open to summary my justification again, but as a tl;dr, I have a
> strong evidence that he knew we never had a consensus about 4.0 which
> destroys his claim for “we agreed to release Spark 4.0.0 as it is”, and
> also he said my proposal is technically correct, so he is objecting himself
> if he is really casting “veto”.
>
> Worth noting that his last post is all about technical justification of
> “his own proposal”, not about technical objection of my proposal. I’m even
> unsure he really intended to cast a vote. I feel like we are overreacting
> but I’m happy to make progress of this at least.
>
> 2025년 3월 16일 (일) 오전 8:44, Mark Hamstra <ma...@gmail.com>님이 작성:
>
>> Quick administrative note: I don't see any reason why this vote should
>> take a long time, so I expect to close the process and tally the votes
>> in not much more than 48 hours.
>>
>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>> >
>> > There has been enough discussion on this topic already, so I think
>> > that an immediate vote on the validity of Dongjoon's technical
>> > justification for his veto of the "Retain migration logic ... in Spark
>> > 4.0.x" proposal is in order. That technical justification has been
>> > called into question, and the guidance at
>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>> > PMC to determine whether the technical justification is  valid: "In
>> > case of doubt, deciding whether a technical justification is valid is
>> > up to the PMC." As such, only PMC votes will decide the outcome of
>> > this vote. This is neither a vote on a code change itself not a vote
>> > on whether a package is ready for release, so it a procedural vote on
>> > whether the technical justification is valid. As such, the vote will
>> > be decided by a simple majority where +1 votes hold that the technical
>> > justification is not valid and -1 votes hold that the technical
>> > justification is valid.
>> >
>> > I would request that at least PMC members post more than just a naked
>> > vote, but instead endeavor to give some reason why they have assessed
>> > the technical justification as they have. I'll start:
>> >
>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>> > confess to still not being entirely clear on what is his technical
>> > justification for that veto. I see claims that including an admonition
>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>> > required to maintain the integrity of already existing data streams,
>> > and I see assertions about the maintenance burden that including the
>> > migration logic would impose on future Spark versions, but I don't
>> > think that I see any other technical objections. I do not believe that
>> > the claimed technical justification is valid.
>> >
>> > In requiring that a veto of a code change be accompanied by a
>> > technical justification for the veto, the Apache Voting Process states
>> > that: "To prevent vetoes from being used capriciously, the voter must
>> > provide with the veto a technical justification showing why the change
>> > is bad (opens a security exposure, negatively affects performance,
>> > etc. ). A veto without a justification is invalid and has no weight."
>> > This strongly implies that there must be something objectively wrong
>> > with the proposed code change in that it causes significant harm in
>> > the way of opening a security exposure, negatively affecting
>> > performance, or presumably other significant user harms or perhaps
>> > even developer burdens.
>> >
>> > The proposed addition of the migration logic to Spark 4.0.x does not
>> > cause any harm to Spark's users. For many users, those not using
>> > streaming data, the change will have no effect. For streaming users
>> > the change will be beneficial, not harmful.
>> >
>> > Neither do I find the claim of excessive, ongoing developer burden to
>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>> > it wouldn't surprise me if no further changes to this migration logic
>> > would be needed for a very long time.
>> >
>> > Some of what we are left with is just an expression of preference for
>> > a technical alternative to the migration logic -- i.e. including in
>> > the release notes an admonition to first upgrade to 3.5.5. But the
>> > Apache Voting Process does not say that in the face of code
>> > alternatives A and B, a qualified voter is justified in vetoing A if
>> > they prefer B. Instead, the Voting Process strongly implies that
>> > something more is needed to justify a veto, as I've already covered.
>> > Thus I don't find Dongjoon's preference for the release notes option
>> > to be adequate justification for the veto.
>> >
>> > The only remaining question I see is whether including "databricks" in
>> > the Apache Code is ever allowed or if any such instance must be
>> > expunged as soon as possible. I am not aware of any ASF policy that
>> > strictly forbids the mention of a vendor in Apache code for any
>> > reason, even if that vendor has a product based on Apache code, even
>> > if that vendor enjoys a uniquely influential position vis a vis some
>> > Apache code or project. Certainly the PMC has a duty to see to it that
>> > neither Databricks nor any other vendor exercises influence or control
>> > over Apache Spark outside of the established Apache process, but the
>> > proposed migration code changes do not advantage Databricks -- if
>> > anything they remove a minor avenue of influence, and simply need to
>> > mention "databricks" once in order match and transform a configuration
>> > into a vendor neutral equivalent. While not optimal, I can't find such
>> > a one-time inclusion of "databricks" to be truly offensive to any
>> > non-technical policy concern -- certainly not offensive to the point
>> > that it outweighs the user advantage of including the migration logic
>> > in Spark 4.0.x.
>> >
>> > In summary, I do not find Dongjoon's given technical justification to
>> > be valid relative to the Apache requirements for a veto of a code
>> > change, so I must vote...
>> >
>> > +1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Slight correction (to avoid nitpick): “vote” in the last sentence is
“veto”. I think community will understand correctly, but just to avoid any
voice making this to be sidetracked.

2025년 3월 16일 (일) 오전 11:20, Jungtaek Lim <ka...@gmail.com>님이 작성:

> For sure, I’m +1 (non-binding).
>
> I believe I don’t need to explain more and I spent whole weekend and we
> have a history about my justification in the history of mailing list.
>
> I’m open to summary my justification again, but as a tl;dr, I have a
> strong evidence that he knew we never had a consensus about 4.0 which
> destroys his claim for “we agreed to release Spark 4.0.0 as it is”, and
> also he said my proposal is technically correct, so he is objecting himself
> if he is really casting “veto”.
>
> Worth noting that his last post is all about technical justification of
> “his own proposal”, not about technical objection of my proposal. I’m even
> unsure he really intended to cast a vote. I feel like we are overreacting
> but I’m happy to make progress of this at least.
>
> 2025년 3월 16일 (일) 오전 8:44, Mark Hamstra <ma...@gmail.com>님이 작성:
>
>> Quick administrative note: I don't see any reason why this vote should
>> take a long time, so I expect to close the process and tally the votes
>> in not much more than 48 hours.
>>
>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>> >
>> > There has been enough discussion on this topic already, so I think
>> > that an immediate vote on the validity of Dongjoon's technical
>> > justification for his veto of the "Retain migration logic ... in Spark
>> > 4.0.x" proposal is in order. That technical justification has been
>> > called into question, and the guidance at
>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>> > PMC to determine whether the technical justification is  valid: "In
>> > case of doubt, deciding whether a technical justification is valid is
>> > up to the PMC." As such, only PMC votes will decide the outcome of
>> > this vote. This is neither a vote on a code change itself not a vote
>> > on whether a package is ready for release, so it a procedural vote on
>> > whether the technical justification is valid. As such, the vote will
>> > be decided by a simple majority where +1 votes hold that the technical
>> > justification is not valid and -1 votes hold that the technical
>> > justification is valid.
>> >
>> > I would request that at least PMC members post more than just a naked
>> > vote, but instead endeavor to give some reason why they have assessed
>> > the technical justification as they have. I'll start:
>> >
>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>> > confess to still not being entirely clear on what is his technical
>> > justification for that veto. I see claims that including an admonition
>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>> > required to maintain the integrity of already existing data streams,
>> > and I see assertions about the maintenance burden that including the
>> > migration logic would impose on future Spark versions, but I don't
>> > think that I see any other technical objections. I do not believe that
>> > the claimed technical justification is valid.
>> >
>> > In requiring that a veto of a code change be accompanied by a
>> > technical justification for the veto, the Apache Voting Process states
>> > that: "To prevent vetoes from being used capriciously, the voter must
>> > provide with the veto a technical justification showing why the change
>> > is bad (opens a security exposure, negatively affects performance,
>> > etc. ). A veto without a justification is invalid and has no weight."
>> > This strongly implies that there must be something objectively wrong
>> > with the proposed code change in that it causes significant harm in
>> > the way of opening a security exposure, negatively affecting
>> > performance, or presumably other significant user harms or perhaps
>> > even developer burdens.
>> >
>> > The proposed addition of the migration logic to Spark 4.0.x does not
>> > cause any harm to Spark's users. For many users, those not using
>> > streaming data, the change will have no effect. For streaming users
>> > the change will be beneficial, not harmful.
>> >
>> > Neither do I find the claim of excessive, ongoing developer burden to
>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>> > it wouldn't surprise me if no further changes to this migration logic
>> > would be needed for a very long time.
>> >
>> > Some of what we are left with is just an expression of preference for
>> > a technical alternative to the migration logic -- i.e. including in
>> > the release notes an admonition to first upgrade to 3.5.5. But the
>> > Apache Voting Process does not say that in the face of code
>> > alternatives A and B, a qualified voter is justified in vetoing A if
>> > they prefer B. Instead, the Voting Process strongly implies that
>> > something more is needed to justify a veto, as I've already covered.
>> > Thus I don't find Dongjoon's preference for the release notes option
>> > to be adequate justification for the veto.
>> >
>> > The only remaining question I see is whether including "databricks" in
>> > the Apache Code is ever allowed or if any such instance must be
>> > expunged as soon as possible. I am not aware of any ASF policy that
>> > strictly forbids the mention of a vendor in Apache code for any
>> > reason, even if that vendor has a product based on Apache code, even
>> > if that vendor enjoys a uniquely influential position vis a vis some
>> > Apache code or project. Certainly the PMC has a duty to see to it that
>> > neither Databricks nor any other vendor exercises influence or control
>> > over Apache Spark outside of the established Apache process, but the
>> > proposed migration code changes do not advantage Databricks -- if
>> > anything they remove a minor avenue of influence, and simply need to
>> > mention "databricks" once in order match and transform a configuration
>> > into a vendor neutral equivalent. While not optimal, I can't find such
>> > a one-time inclusion of "databricks" to be truly offensive to any
>> > non-technical policy concern -- certainly not offensive to the point
>> > that it outweighs the user advantage of including the migration logic
>> > in Spark 4.0.x.
>> >
>> > In summary, I do not find Dongjoon's given technical justification to
>> > be valid relative to the Apache requirements for a veto of a code
>> > change, so I must vote...
>> >
>> > +1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

For sure, I’m +1 (non-binding).

I believe I don’t need to explain more and I spent whole weekend and we
have a history about my justification in the history of mailing list.

I’m open to summary my justification again, but as a tl;dr, I have a strong
evidence that he knew we never had a consensus about 4.0 which destroys his
claim for “we agreed to release Spark 4.0.0 as it is”, and also he said my
proposal is technically correct, so he is objecting himself if he is really
casting “veto”.

Worth noting that his last post is all about technical justification of
“his own proposal”, not about technical objection of my proposal. I’m even
unsure he really intended to cast a vote. I feel like we are overreacting
but I’m happy to make progress of this at least.

2025년 3월 16일 (일) 오전 8:44, Mark Hamstra <ma...@gmail.com>님이 작성:

> Quick administrative note: I don't see any reason why this vote should
> take a long time, so I expect to close the process and tally the votes
> in not much more than 48 hours.
>
> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
> wrote:
> >
> > There has been enough discussion on this topic already, so I think
> > that an immediate vote on the validity of Dongjoon's technical
> > justification for his veto of the "Retain migration logic ... in Spark
> > 4.0.x" proposal is in order. That technical justification has been
> > called into question, and the guidance at
> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
> > PMC to determine whether the technical justification is  valid: "In
> > case of doubt, deciding whether a technical justification is valid is
> > up to the PMC." As such, only PMC votes will decide the outcome of
> > this vote. This is neither a vote on a code change itself not a vote
> > on whether a package is ready for release, so it a procedural vote on
> > whether the technical justification is valid. As such, the vote will
> > be decided by a simple majority where +1 votes hold that the technical
> > justification is not valid and -1 votes hold that the technical
> > justification is valid.
> >
> > I would request that at least PMC members post more than just a naked
> > vote, but instead endeavor to give some reason why they have assessed
> > the technical justification as they have. I'll start:
> >
> > Despite all of the discussion related to Dongjoon's -1 vote, I must
> > confess to still not being entirely clear on what is his technical
> > justification for that veto. I see claims that including an admonition
> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
> > required to maintain the integrity of already existing data streams,
> > and I see assertions about the maintenance burden that including the
> > migration logic would impose on future Spark versions, but I don't
> > think that I see any other technical objections. I do not believe that
> > the claimed technical justification is valid.
> >
> > In requiring that a veto of a code change be accompanied by a
> > technical justification for the veto, the Apache Voting Process states
> > that: "To prevent vetoes from being used capriciously, the voter must
> > provide with the veto a technical justification showing why the change
> > is bad (opens a security exposure, negatively affects performance,
> > etc. ). A veto without a justification is invalid and has no weight."
> > This strongly implies that there must be something objectively wrong
> > with the proposed code change in that it causes significant harm in
> > the way of opening a security exposure, negatively affecting
> > performance, or presumably other significant user harms or perhaps
> > even developer burdens.
> >
> > The proposed addition of the migration logic to Spark 4.0.x does not
> > cause any harm to Spark's users. For many users, those not using
> > streaming data, the change will have no effect. For streaming users
> > the change will be beneficial, not harmful.
> >
> > Neither do I find the claim of excessive, ongoing developer burden to
> > be persuasive. The changes are tiny and easily maintained -- in fact,
> > it wouldn't surprise me if no further changes to this migration logic
> > would be needed for a very long time.
> >
> > Some of what we are left with is just an expression of preference for
> > a technical alternative to the migration logic -- i.e. including in
> > the release notes an admonition to first upgrade to 3.5.5. But the
> > Apache Voting Process does not say that in the face of code
> > alternatives A and B, a qualified voter is justified in vetoing A if
> > they prefer B. Instead, the Voting Process strongly implies that
> > something more is needed to justify a veto, as I've already covered.
> > Thus I don't find Dongjoon's preference for the release notes option
> > to be adequate justification for the veto.
> >
> > The only remaining question I see is whether including "databricks" in
> > the Apache Code is ever allowed or if any such instance must be
> > expunged as soon as possible. I am not aware of any ASF policy that
> > strictly forbids the mention of a vendor in Apache code for any
> > reason, even if that vendor has a product based on Apache code, even
> > if that vendor enjoys a uniquely influential position vis a vis some
> > Apache code or project. Certainly the PMC has a duty to see to it that
> > neither Databricks nor any other vendor exercises influence or control
> > over Apache Spark outside of the established Apache process, but the
> > proposed migration code changes do not advantage Databricks -- if
> > anything they remove a minor avenue of influence, and simply need to
> > mention "databricks" once in order match and transform a configuration
> > into a vendor neutral equivalent. While not optimal, I can't find such
> > a one-time inclusion of "databricks" to be truly offensive to any
> > non-technical policy concern -- certainly not offensive to the point
> > that it outweighs the user advantage of including the migration logic
> > in Spark 4.0.x.
> >
> > In summary, I do not find Dongjoon's given technical justification to
> > be valid relative to the Apache requirements for a veto of a code
> > change, so I must vote...
> >
> > +1
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Regardless of this vote result, we (including me) had too many
mistakes/faults during handling this (I'd rather say "incident", or even
"disaster").

I really think we should have a retrospective, making a rough timeline of
the events with making people involved to be anonymous, and brainstorm how
we could improve the mindset / process / etc (even BYLAW) to avoid this
from happening again. Personally, I was seriously considering escalating
this based on the events that happened to me from this incident. But
instead of doing that, discussing in public and thinking about how we could
prevent bad things from happening again, sounds to me a lot beneficial to
everyone, because anyone can be "someone" in the situation. I mean, this is
the last time I am open to finding a better solution, and if this happens
again I am willing to immediately escalate the problem.

For sure, it's not perfect and there is zero way to completely anonymize
this (someone might be willing to waste their time to correlate the
anonymized event and the real event), but if we don't do that because we
might indirectly point out someone, I think we will never be able to bring
the problem the community and individual make to the public discussion, and
then, the only way to resolve this is to escalate which directly jumps to
the last resort. Writing a postmortem is like that - the content has to
call out people, but the mindset is just to brainstorm how we can prevent
this from happening again. If the postmortem is too heavy, maybe just call
it a retrospective.

I understand people being involved in this event (including me) might just
want to forget about it and move on, but arguably this event was a perfect
example of how things could get messed up and make some people
to lose faith of someone/community, and we should revisit what we did and
be sure not to do that again. Otherwise, this will definitely happen again.

On Tue, Mar 18, 2025 at 7:27 AM Hyukjin Kwon <gu...@apache.org> wrote:

> I would like to conclude this vote with my own -1, for the same reasons as
> Holden.
> Personally, I feel that this approach has been overly excessive in several
> ways - such as overriding a veto - and it would have been better to explore
> alternative solutions first.
> That said, it seems that this vote has passed by a simple majority, and
> the vote in the other thread remains valid.
> Since it has now been 48 hours, as mentioned earlier in the thread, I hope
> we can formally conclude this vote.
>
> On Tue, 18 Mar 2025 at 06:47, Yuanjian Li <xy...@gmail.com> wrote:
>
>> +1
>>
>> I would like to add my perspective to this discussion. It's important
>> that we maintain a fair and objective technical evaluation process,
>> especially when a veto is cast. A veto should be backed by concrete
>> technical reasoning.
>>
>> From my reading of the discussion, the key concern is whether the
>> justification for the veto is based on an objective technical issue with
>> the proposed change. If the concern is not rooted in a clear technical
>> limitation or regression, then it should be open to further discussion and
>> reconsideration. It is crucial for us as a community to ensure that
>> decisions are made transparently and with technical merit as the primary
>> driver.
>>
>> I also want to emphasize that maintaining a welcoming and inclusive
>> community is vital to the success of Apache Spark. If contributors feel
>> isolated or that decisions are being swayed by affiliations rather than
>> technical arguments, that would set a concerning precedent. We should
>> strive to engage in discussions constructively and focus on what is best
>> for the project and its users.
>>
>> Jungtaek Lim <ka...@gmail.com> 于2025年3月16日周日 23:11写道：
>>
>>> The problem you are mentioning is arguably resolvable if we leave a
>>> config name as it is, and add
>>> withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan").
>>> Let's not nitpick just from reverting the PR. We have to revert the PR
>>> "semantically".
>>>
>>> Btw, from what I understand of dealing with backport PR is, we mostly
>>> merge in master to begin with, and down the version line. When we handle my
>>> migration PR, we do not follow this practice without any discussion. I
>>> submitted PRs for master/4.0/3.5, and 3.5 was merged "first", and when I
>>> asked for merging to 4.0, I was pushed back. I don't think this is an
>>> ordinary practice of dealing with multiple versions of PRs, especially
>>> since the author never agreed with the way of processing.
>>>
>>> If we were following the practice, we should already have migration
>>> logic for master/4.0/3.5, since we should have the same fix in master/4.0.
>>> We probably had a discussion about removing config from master/4.0 based on
>>> the discussion, and we probably agreed to remove the config since we still
>>> have a migration logic. W.r.t migration logic, based on the discussion we
>>> are having, we probably can't make an agreement to take it out, then
>>> arguably the migration logic is left as it is.
>>>
>>> This way I never needed to drive such a long and sensitive DISCUSSION
>>> and VOTE. But the PR for master and 4.0 weren't merged because of the
>>> individual's belief of the rollout plan, which we are seeing does not fit a
>>> majority of voices now.
>>>
>>> Shall we fix the broken process in which we made a huge mistake before
>>> moving on? We should have merged the same content in 3.5 to master/4.0 as
>>> well, and then have a PR to remove the config. This is totally swapped
>>> which does not make sense to me.
>>>
>>> On Mon, Mar 17, 2025 at 1:57 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> I reviewed Wenchen's reverting PR. Although it's a proposal for
>>>> discussion, it is another breaking change against Apache Spark 3.5.5, isn't
>>>> it? If we consider Apache Spark 3.5.4 users, I believe we need to consider
>>>> Apache Spark 3.5.5 users too which uses
>>>> `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.
>>>>
>>>> https://github.com/apache/spark/pull/50291
>>>>
>>>> - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>>> +
>>>> buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>>>
>>>> Dongjoon.
>>>>
>>>> On 2025/03/17 02:36:42 Wenchen Fan wrote:
>>>> > I've created the revert PR for branch-4.0:
>>>> > https://github.com/apache/spark/pull/50291 . We can merge PRs with
>>>> lazy
>>>> > consensus but it's clear that this breaking change PR has failed to
>>>> achieve
>>>> > consensus.
>>>> >
>>>> > I hope we now have a clear foundation for discussing solutions. As it
>>>> > stands, the misnamed configuration will be released in 4.0.0. I like
>>>> > Jungtaek’s proposal to deprecate it, but the decision is up to the
>>>> > community.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Hyukjin Kwon <gu...@apache.org>]

I would like to conclude this vote with my own -1, for the same reasons as
Holden.
Personally, I feel that this approach has been overly excessive in several
ways - such as overriding a veto - and it would have been better to explore
alternative solutions first.
That said, it seems that this vote has passed by a simple majority, and the
vote in the other thread remains valid.
Since it has now been 48 hours, as mentioned earlier in the thread, I hope
we can formally conclude this vote.

On Tue, 18 Mar 2025 at 06:47, Yuanjian Li <xy...@gmail.com> wrote:

> +1
>
> I would like to add my perspective to this discussion. It's important that
> we maintain a fair and objective technical evaluation process, especially
> when a veto is cast. A veto should be backed by concrete technical
> reasoning.
>
> From my reading of the discussion, the key concern is whether the
> justification for the veto is based on an objective technical issue with
> the proposed change. If the concern is not rooted in a clear technical
> limitation or regression, then it should be open to further discussion and
> reconsideration. It is crucial for us as a community to ensure that
> decisions are made transparently and with technical merit as the primary
> driver.
>
> I also want to emphasize that maintaining a welcoming and inclusive
> community is vital to the success of Apache Spark. If contributors feel
> isolated or that decisions are being swayed by affiliations rather than
> technical arguments, that would set a concerning precedent. We should
> strive to engage in discussions constructively and focus on what is best
> for the project and its users.
>
> Jungtaek Lim <ka...@gmail.com> 于2025年3月16日周日 23:11写道：
>
>> The problem you are mentioning is arguably resolvable if we leave a
>> config name as it is, and add
>> withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan").
>> Let's not nitpick just from reverting the PR. We have to revert the PR
>> "semantically".
>>
>> Btw, from what I understand of dealing with backport PR is, we mostly
>> merge in master to begin with, and down the version line. When we handle my
>> migration PR, we do not follow this practice without any discussion. I
>> submitted PRs for master/4.0/3.5, and 3.5 was merged "first", and when I
>> asked for merging to 4.0, I was pushed back. I don't think this is an
>> ordinary practice of dealing with multiple versions of PRs, especially
>> since the author never agreed with the way of processing.
>>
>> If we were following the practice, we should already have migration logic
>> for master/4.0/3.5, since we should have the same fix in master/4.0. We
>> probably had a discussion about removing config from master/4.0 based on
>> the discussion, and we probably agreed to remove the config since we still
>> have a migration logic. W.r.t migration logic, based on the discussion we
>> are having, we probably can't make an agreement to take it out, then
>> arguably the migration logic is left as it is.
>>
>> This way I never needed to drive such a long and sensitive DISCUSSION and
>> VOTE. But the PR for master and 4.0 weren't merged because of the
>> individual's belief of the rollout plan, which we are seeing does not fit a
>> majority of voices now.
>>
>> Shall we fix the broken process in which we made a huge mistake before
>> moving on? We should have merged the same content in 3.5 to master/4.0 as
>> well, and then have a PR to remove the config. This is totally swapped
>> which does not make sense to me.
>>
>> On Mon, Mar 17, 2025 at 1:57 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> I reviewed Wenchen's reverting PR. Although it's a proposal for
>>> discussion, it is another breaking change against Apache Spark 3.5.5, isn't
>>> it? If we consider Apache Spark 3.5.4 users, I believe we need to consider
>>> Apache Spark 3.5.5 users too which uses
>>> `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.
>>>
>>> https://github.com/apache/spark/pull/50291
>>>
>>> - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>> +
>>> buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>>
>>> Dongjoon.
>>>
>>> On 2025/03/17 02:36:42 Wenchen Fan wrote:
>>> > I've created the revert PR for branch-4.0:
>>> > https://github.com/apache/spark/pull/50291 . We can merge PRs with
>>> lazy
>>> > consensus but it's clear that this breaking change PR has failed to
>>> achieve
>>> > consensus.
>>> >
>>> > I hope we now have a clear foundation for discussing solutions. As it
>>> > stands, the misnamed configuration will be released in 4.0.0. I like
>>> > Jungtaek’s proposal to deprecate it, but the decision is up to the
>>> > community.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Yuanjian Li <xy...@gmail.com>]

+1

I would like to add my perspective to this discussion. It's important that
we maintain a fair and objective technical evaluation process, especially
when a veto is cast. A veto should be backed by concrete technical
reasoning.

From my reading of the discussion, the key concern is whether the
justification for the veto is based on an objective technical issue with
the proposed change. If the concern is not rooted in a clear technical
limitation or regression, then it should be open to further discussion and
reconsideration. It is crucial for us as a community to ensure that
decisions are made transparently and with technical merit as the primary
driver.

I also want to emphasize that maintaining a welcoming and inclusive
community is vital to the success of Apache Spark. If contributors feel
isolated or that decisions are being swayed by affiliations rather than
technical arguments, that would set a concerning precedent. We should
strive to engage in discussions constructively and focus on what is best
for the project and its users.

Jungtaek Lim <ka...@gmail.com> 于2025年3月16日周日 23:11写道：

> The problem you are mentioning is arguably resolvable if we leave a config
> name as it is, and add
> withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan").
> Let's not nitpick just from reverting the PR. We have to revert the PR
> "semantically".
>
> Btw, from what I understand of dealing with backport PR is, we mostly
> merge in master to begin with, and down the version line. When we handle my
> migration PR, we do not follow this practice without any discussion. I
> submitted PRs for master/4.0/3.5, and 3.5 was merged "first", and when I
> asked for merging to 4.0, I was pushed back. I don't think this is an
> ordinary practice of dealing with multiple versions of PRs, especially
> since the author never agreed with the way of processing.
>
> If we were following the practice, we should already have migration logic
> for master/4.0/3.5, since we should have the same fix in master/4.0. We
> probably had a discussion about removing config from master/4.0 based on
> the discussion, and we probably agreed to remove the config since we still
> have a migration logic. W.r.t migration logic, based on the discussion we
> are having, we probably can't make an agreement to take it out, then
> arguably the migration logic is left as it is.
>
> This way I never needed to drive such a long and sensitive DISCUSSION and
> VOTE. But the PR for master and 4.0 weren't merged because of the
> individual's belief of the rollout plan, which we are seeing does not fit a
> majority of voices now.
>
> Shall we fix the broken process in which we made a huge mistake before
> moving on? We should have merged the same content in 3.5 to master/4.0 as
> well, and then have a PR to remove the config. This is totally swapped
> which does not make sense to me.
>
> On Mon, Mar 17, 2025 at 1:57 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> I reviewed Wenchen's reverting PR. Although it's a proposal for
>> discussion, it is another breaking change against Apache Spark 3.5.5, isn't
>> it? If we consider Apache Spark 3.5.4 users, I believe we need to consider
>> Apache Spark 3.5.5 users too which uses
>> `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.
>>
>> https://github.com/apache/spark/pull/50291
>>
>> - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>> +
>> buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>
>> Dongjoon.
>>
>> On 2025/03/17 02:36:42 Wenchen Fan wrote:
>> > I've created the revert PR for branch-4.0:
>> > https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
>> > consensus but it's clear that this breaking change PR has failed to
>> achieve
>> > consensus.
>> >
>> > I hope we now have a clear foundation for discussing solutions. As it
>> > stands, the misnamed configuration will be released in 4.0.0. I like
>> > Jungtaek’s proposal to deprecate it, but the decision is up to the
>> > community.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

The problem you are mentioning is arguably resolvable if we leave a config
name as it is, and add
withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan").
Let's not nitpick just from reverting the PR. We have to revert the PR
"semantically".

Btw, from what I understand of dealing with backport PR is, we mostly merge
in master to begin with, and down the version line. When we handle my
migration PR, we do not follow this practice without any discussion. I
submitted PRs for master/4.0/3.5, and 3.5 was merged "first", and when I
asked for merging to 4.0, I was pushed back. I don't think this is an
ordinary practice of dealing with multiple versions of PRs, especially
since the author never agreed with the way of processing.

If we were following the practice, we should already have migration logic
for master/4.0/3.5, since we should have the same fix in master/4.0. We
probably had a discussion about removing config from master/4.0 based on
the discussion, and we probably agreed to remove the config since we still
have a migration logic. W.r.t migration logic, based on the discussion we
are having, we probably can't make an agreement to take it out, then
arguably the migration logic is left as it is.

This way I never needed to drive such a long and sensitive DISCUSSION and
VOTE. But the PR for master and 4.0 weren't merged because of the
individual's belief of the rollout plan, which we are seeing does not fit a
majority of voices now.

Shall we fix the broken process in which we made a huge mistake before
moving on? We should have merged the same content in 3.5 to master/4.0 as
well, and then have a PR to remove the config. This is totally swapped
which does not make sense to me.

On Mon, Mar 17, 2025 at 1:57 PM Dongjoon Hyun <do...@apache.org> wrote:

> I reviewed Wenchen's reverting PR. Although it's a proposal for
> discussion, it is another breaking change against Apache Spark 3.5.5, isn't
> it? If we consider Apache Spark 3.5.4 users, I believe we need to consider
> Apache Spark 3.5.5 users too which uses
> `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.
>
> https://github.com/apache/spark/pull/50291
>
> - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
> +
> buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>
> Dongjoon.
>
> On 2025/03/17 02:36:42 Wenchen Fan wrote:
> > I've created the revert PR for branch-4.0:
> > https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
> > consensus but it's clear that this breaking change PR has failed to
> achieve
> > consensus.
> >
> > I hope we now have a clear foundation for discussing solutions. As it
> > stands, the misnamed configuration will be released in 4.0.0. I like
> > Jungtaek’s proposal to deprecate it, but the decision is up to the
> > community.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Dongjoon Hyun <do...@apache.org>]

I reviewed Wenchen's reverting PR. Although it's a proposal for discussion, it is another breaking change against Apache Spark 3.5.5, isn't it? If we consider Apache Spark 3.5.4 users, I believe we need to consider Apache Spark 3.5.5 users too which uses `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.

https://github.com/apache/spark/pull/50291

- buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
+ buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")

Dongjoon.

On 2025/03/17 02:36:42 Wenchen Fan wrote:
> I've created the revert PR for branch-4.0:
> https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
> consensus but it's clear that this breaking change PR has failed to achieve
> consensus.
> 
> I hope we now have a clear foundation for discussing solutions. As it
> stands, the misnamed configuration will be released in 4.0.0. I like
> Jungtaek’s proposal to deprecate it, but the decision is up to the
> community.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Btw, I don't think reverting the PR of removing config is violating
community's consensus, because we never had a VOTE for this breaking change
(this is definitely a breaking change without migration logic), so we did
not really get explicit consensus with that. The real neutral state is that
we do not have a code change for removing config. (I'm open to hear
objections.)

It was merged based on the assumption that we will figure out how to NOT
break backward compatibility, and since this is NOT addressed, I argue the
change of removing config has to be reverted. Even further, this does not
violate the community's consensus since I have strong evidence of VETO on
that PR - We generally do not consider breaking change without any
alternative to be valid change.

I'm happy to go and cast -1 to that PR (committer should have been
considered to have a binding vote in code, otherwise my +1 cannot merge the
PR), and revert unless anyone objects with valid reason.

On Mon, Mar 17, 2025 at 12:31 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Holden, I think I have some workaround like I posted in dev@ (link
> <https://lists.apache.org/thread/xsq58800smtc5xo15kfzyj5kfw5yh7wj>),
> which is definitely less better than this proposal so I still want to see
> this VOTE to go forward, but it's somewhat better in this situation that we
> no longer talk about vendor name, hence no need to debate for more minor
> versions.
>
> Though I still want to hear the community's voice that "Are we really
> thinking that forcing users to take the upgrade path as we guided is making
> sense?". I'm not strongly arguing that the vendor name has to be in the
> codebase (but if this is easier way then we should weigh), but I strongly
> argue that we have no control of users, so we must always think how we can
> achieve letting users do whatever they want, with minimized inconvenience
> (since we can't always make things not break). I think this meta question
> is a strong argument from me, and I'm -1 to Dongjoon's proposal because his
> proposal is breaking this.
>
> I strongly believe there is NO way to force users to upgrade to a specific
> version - I think we should have sought alternatives when this came up with
> the main proposal.
>
> 2025년 3월 17일 (월) 오후 12:10, Holden Karau <ho...@gmail.com>님이 작성:
>
>> I'm delighted to see folks talking about a compromise. However, instead
>> of just asking Dongjoon to withdraw the VETO perhaps folks can suggest
>> alternatives that that might meet some of both parties goals?
>>
>> On Sun, Mar 16, 2025 at 7:41 PM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> I agree with Holden that withdrawing a veto is always better than
>>> overriding it: it's healthier for the community. Dongjoon, would you be
>>> willing to reconsider your veto given the current as-is state of the 4.0.0
>>> release (the breaking change will be reverted)?
>>>
>>> On Mon, Mar 17, 2025 at 10:36 AM Wenchen Fan <cl...@gmail.com>
>>> wrote:
>>>
>>>> I've created the revert PR for branch-4.0:
>>>> https://github.com/apache/spark/pull/50291 . We can merge PRs with
>>>> lazy consensus but it's clear that this breaking change PR has failed to
>>>> achieve consensus.
>>>>
>>>> I hope we now have a clear foundation for discussing solutions. As it
>>>> stands, the misnamed configuration will be released in 4.0.0. I like
>>>> Jungtaek’s proposal to deprecate it, but the decision is up to the
>>>> community.
>>>>
>>>> On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>>
>>>>> OK, let's be super honest.
>>>>>
>>>>> Again, I think you agree that *"both" proposals are "technically"
>>>>> correct (or one side can't have a strong theoretical evidence to counter
>>>>> the other side)*. So this naturally has a fate to have more
>>>>> supporters to get to the end. It's very easy for me to VETO to his proposal
>>>>> (although I don't have a binding vote, I think I have people who agree with
>>>>> me) if we think we want to definitely expand the interpretation of VETO
>>>>> criteria in the Apache Voting Process.
>>>>>
>>>>> You said it is up to the PMC member exercising the veto to use their
>>>>> judgement, but definitely, it must not be used to force the community to
>>>>> follow his proposal. The major argument here is, he can just VETO to any
>>>>> proposal to retain the codebase as the way he prefers to, which I don't
>>>>> believe is a correct usage of VETO.
>>>>>
>>>>> If we just revert the change of removal of config, this is "really"
>>>>> neutral neither my proposal nor his proposal. Do we really want to do so?
>>>>>
>>>>> On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <ho...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> First let me start with my key hope:
>>>>>>
>>>>>> We find a way to compromise and have the veto withdrawn rather than
>>>>>> overridden.
>>>>>>
>>>>>> From what I understand of the change in question:
>>>>>>
>>>>>> So my understanding, and I may be over simplifying here but there are
>>>>>> (at least) three technical paths forward (migration guide, legacy config
>>>>>> with vendor string in it, non-vendor specific string legacy config), a PMC
>>>>>> member vetoed one of them (named vendor legacy config) because he thought a
>>>>>> different approach was better (migration guide) as they were worried that
>>>>>> carrying that legacy config forward would encourage bad coding standards
>>>>>> (eg we would add more vendor named config flags). To me that seems like a
>>>>>> valid concern.
>>>>>>
>>>>>> My reasoning:
>>>>>>
>>>>>> Thinking back at other VETOs that I’ve been involved with in this
>>>>>> project (DSV2, graceful decom, etc) this seems to meet the same bar. Hell
>>>>>> we’ve had plenty of vetos that didn’t offer an alternative.
>>>>>>
>>>>>> My personal understanding of where the bar for “
>>>>>> a technical justification showing why the change is bad” concern is
>>>>>> pretty much “any not factually incorrect reasoning”, the text doesn’t have
>>>>>> any particular “bar” for the level of “badness” and I think it’s up to the
>>>>>> PMC member exercising the veto to use their judgement.
>>>>>>
>>>>>> In closing, I feel like the path we’re going down (overriding a veto)
>>>>>> is not healthy for the project.
>>>>>>
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>> Pronouns: she/her
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <
>>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>>
>>>>>>> Holden, I believe you should already know "both" approaches are
>>>>>>> "technically" correct. It's not about which one you have a preference for,
>>>>>>> no, this VOTE is not intended to extend the debate.
>>>>>>>
>>>>>>> Again, what you are encouraged to do here is, not exposing your
>>>>>>> preference of two approaches, but exposing your "technically valid" concern
>>>>>>> of my approach, backed by Dongjoon's veto (most likely you want to quote
>>>>>>> Dongjoon's post). This is very simple and I'm not sure you are doing
>>>>>>> exactly what the VOTE requires.
>>>>>>>
>>>>>>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>>>>>>> objection is technical then I think we need to respect the veto. There is a
>>>>>>>> fundamental disagreement as to what the correct technical way to address
>>>>>>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>>>>>>> has vetoed  the legacy config option.
>>>>>>>>
>>>>>>>> I think I disagree with Mark on the assertion that the veto needs
>>>>>>>> to have “substantial technical concern,” but rather a valid concern. I
>>>>>>>> think in addition to the veto they’ve also gone above and beyond providing
>>>>>>>> alternative ways to accomplish this.
>>>>>>>>
>>>>>>>> On a personal level:
>>>>>>>>
>>>>>>>> I am optimistic we can unblock the release but I think it’s
>>>>>>>> important to err on the side of respecting the veto here in the interest of
>>>>>>>> perceived fairness *especially* because of vendor aspects.
>>>>>>>>
>>>>>>>> To be clear I’ve worked at most of these companies (and many of the
>>>>>>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>>>>>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>>>>>>> here and would ask that we show everyone understanding regardless of the
>>>>>>>> outcome.
>>>>>>>>
>>>>>>>> Sending hugs and good vibes to y’all.
>>>>>>>>
>>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>> Pronouns: she/her
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <
>>>>>>>> holden.karau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Given it’s the weekend maybe let’s give folks at least one full
>>>>>>>>> work day.
>>>>>>>>>
>>>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>>> Pronouns: she/her
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <
>>>>>>>>> markhamstra@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Quick administrative note: I don't see any reason why this vote
>>>>>>>>>> should
>>>>>>>>>> take a long time, so I expect to close the process and tally the
>>>>>>>>>> votes
>>>>>>>>>> in not much more than 48 hours.
>>>>>>>>>>
>>>>>>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <
>>>>>>>>>> markhamstra@gmail.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> > There has been enough discussion on this topic already, so I
>>>>>>>>>> think
>>>>>>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>>>>>>> > justification for his veto of the "Retain migration logic ...
>>>>>>>>>> in Spark
>>>>>>>>>> > 4.0.x" proposal is in order. That technical justification has
>>>>>>>>>> been
>>>>>>>>>> > called into question, and the guidance at
>>>>>>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it
>>>>>>>>>> to the
>>>>>>>>>> > PMC to determine whether the technical justification is  valid:
>>>>>>>>>> "In
>>>>>>>>>> > case of doubt, deciding whether a technical justification is
>>>>>>>>>> valid is
>>>>>>>>>> > up to the PMC." As such, only PMC votes will decide the outcome
>>>>>>>>>> of
>>>>>>>>>> > this vote. This is neither a vote on a code change itself not a
>>>>>>>>>> vote
>>>>>>>>>> > on whether a package is ready for release, so it a procedural
>>>>>>>>>> vote on
>>>>>>>>>> > whether the technical justification is valid. As such, the vote
>>>>>>>>>> will
>>>>>>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>>>>>>> technical
>>>>>>>>>> > justification is not valid and -1 votes hold that the technical
>>>>>>>>>> > justification is valid.
>>>>>>>>>> >
>>>>>>>>>> > I would request that at least PMC members post more than just a
>>>>>>>>>> naked
>>>>>>>>>> > vote, but instead endeavor to give some reason why they have
>>>>>>>>>> assessed
>>>>>>>>>> > the technical justification as they have. I'll start:
>>>>>>>>>> >
>>>>>>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I
>>>>>>>>>> must
>>>>>>>>>> > confess to still not being entirely clear on what is his
>>>>>>>>>> technical
>>>>>>>>>> > justification for that veto. I see claims that including an
>>>>>>>>>> admonition
>>>>>>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5
>>>>>>>>>> is
>>>>>>>>>> > required to maintain the integrity of already existing data
>>>>>>>>>> streams,
>>>>>>>>>> > and I see assertions about the maintenance burden that
>>>>>>>>>> including the
>>>>>>>>>> > migration logic would impose on future Spark versions, but I
>>>>>>>>>> don't
>>>>>>>>>> > think that I see any other technical objections. I do not
>>>>>>>>>> believe that
>>>>>>>>>> > the claimed technical justification is valid.
>>>>>>>>>> >
>>>>>>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>>>>>>> > technical justification for the veto, the Apache Voting Process
>>>>>>>>>> states
>>>>>>>>>> > that: "To prevent vetoes from being used capriciously, the
>>>>>>>>>> voter must
>>>>>>>>>> > provide with the veto a technical justification showing why the
>>>>>>>>>> change
>>>>>>>>>> > is bad (opens a security exposure, negatively affects
>>>>>>>>>> performance,
>>>>>>>>>> > etc. ). A veto without a justification is invalid and has no
>>>>>>>>>> weight."
>>>>>>>>>> > This strongly implies that there must be something objectively
>>>>>>>>>> wrong
>>>>>>>>>> > with the proposed code change in that it causes significant
>>>>>>>>>> harm in
>>>>>>>>>> > the way of opening a security exposure, negatively affecting
>>>>>>>>>> > performance, or presumably other significant user harms or
>>>>>>>>>> perhaps
>>>>>>>>>> > even developer burdens.
>>>>>>>>>> >
>>>>>>>>>> > The proposed addition of the migration logic to Spark 4.0.x
>>>>>>>>>> does not
>>>>>>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>>>>>>> > streaming data, the change will have no effect. For streaming
>>>>>>>>>> users
>>>>>>>>>> > the change will be beneficial, not harmful.
>>>>>>>>>> >
>>>>>>>>>> > Neither do I find the claim of excessive, ongoing developer
>>>>>>>>>> burden to
>>>>>>>>>> > be persuasive. The changes are tiny and easily maintained -- in
>>>>>>>>>> fact,
>>>>>>>>>> > it wouldn't surprise me if no further changes to this migration
>>>>>>>>>> logic
>>>>>>>>>> > would be needed for a very long time.
>>>>>>>>>> >
>>>>>>>>>> > Some of what we are left with is just an expression of
>>>>>>>>>> preference for
>>>>>>>>>> > a technical alternative to the migration logic -- i.e.
>>>>>>>>>> including in
>>>>>>>>>> > the release notes an admonition to first upgrade to 3.5.5. But
>>>>>>>>>> the
>>>>>>>>>> > Apache Voting Process does not say that in the face of code
>>>>>>>>>> > alternatives A and B, a qualified voter is justified in vetoing
>>>>>>>>>> A if
>>>>>>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>>>>>>> > something more is needed to justify a veto, as I've already
>>>>>>>>>> covered.
>>>>>>>>>> > Thus I don't find Dongjoon's preference for the release notes
>>>>>>>>>> option
>>>>>>>>>> > to be adequate justification for the veto.
>>>>>>>>>> >
>>>>>>>>>> > The only remaining question I see is whether including
>>>>>>>>>> "databricks" in
>>>>>>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>>>>>>> > expunged as soon as possible. I am not aware of any ASF policy
>>>>>>>>>> that
>>>>>>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>>>>>>> > reason, even if that vendor has a product based on Apache code,
>>>>>>>>>> even
>>>>>>>>>> > if that vendor enjoys a uniquely influential position vis a vis
>>>>>>>>>> some
>>>>>>>>>> > Apache code or project. Certainly the PMC has a duty to see to
>>>>>>>>>> it that
>>>>>>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>>>>>>> control
>>>>>>>>>> > over Apache Spark outside of the established Apache process,
>>>>>>>>>> but the
>>>>>>>>>> > proposed migration code changes do not advantage Databricks --
>>>>>>>>>> if
>>>>>>>>>> > anything they remove a minor avenue of influence, and simply
>>>>>>>>>> need to
>>>>>>>>>> > mention "databricks" once in order match and transform a
>>>>>>>>>> configuration
>>>>>>>>>> > into a vendor neutral equivalent. While not optimal, I can't
>>>>>>>>>> find such
>>>>>>>>>> > a one-time inclusion of "databricks" to be truly offensive to
>>>>>>>>>> any
>>>>>>>>>> > non-technical policy concern -- certainly not offensive to the
>>>>>>>>>> point
>>>>>>>>>> > that it outweighs the user advantage of including the migration
>>>>>>>>>> logic
>>>>>>>>>> > in Spark 4.0.x.
>>>>>>>>>> >
>>>>>>>>>> > In summary, I do not find Dongjoon's given technical
>>>>>>>>>> justification to
>>>>>>>>>> > be valid relative to the Apache requirements for a veto of a
>>>>>>>>>> code
>>>>>>>>>> > change, so I must vote...
>>>>>>>>>> >
>>>>>>>>>> > +1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Holden, I think I have some workaround like I posted in dev@ (link
<https://lists.apache.org/thread/xsq58800smtc5xo15kfzyj5kfw5yh7wj>), which
is definitely less better than this proposal so I still want to see this
VOTE to go forward, but it's somewhat better in this situation that we no
longer talk about vendor name, hence no need to debate for more minor
versions.

Though I still want to hear the community's voice that "Are we really
thinking that forcing users to take the upgrade path as we guided is making
sense?". I'm not strongly arguing that the vendor name has to be in the
codebase (but if this is easier way then we should weigh), but I strongly
argue that we have no control of users, so we must always think how we can
achieve letting users do whatever they want, with minimized inconvenience
(since we can't always make things not break). I think this meta question
is a strong argument from me, and I'm -1 to Dongjoon's proposal because his
proposal is breaking this.

I strongly believe there is NO way to force users to upgrade to a specific
version - I think we should have sought alternatives when this came up with
the main proposal.

2025년 3월 17일 (월) 오후 12:10, Holden Karau <ho...@gmail.com>님이 작성:

> I'm delighted to see folks talking about a compromise. However, instead of
> just asking Dongjoon to withdraw the VETO perhaps folks can suggest
> alternatives that that might meet some of both parties goals?
>
> On Sun, Mar 16, 2025 at 7:41 PM Wenchen Fan <cl...@gmail.com> wrote:
>
>> I agree with Holden that withdrawing a veto is always better than
>> overriding it: it's healthier for the community. Dongjoon, would you be
>> willing to reconsider your veto given the current as-is state of the 4.0.0
>> release (the breaking change will be reverted)?
>>
>> On Mon, Mar 17, 2025 at 10:36 AM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> I've created the revert PR for branch-4.0:
>>> https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
>>> consensus but it's clear that this breaking change PR has failed to achieve
>>> consensus.
>>>
>>> I hope we now have a clear foundation for discussing solutions. As it
>>> stands, the misnamed configuration will be released in 4.0.0. I like
>>> Jungtaek’s proposal to deprecate it, but the decision is up to the
>>> community.
>>>
>>> On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> OK, let's be super honest.
>>>>
>>>> Again, I think you agree that *"both" proposals are "technically"
>>>> correct (or one side can't have a strong theoretical evidence to counter
>>>> the other side)*. So this naturally has a fate to have more supporters
>>>> to get to the end. It's very easy for me to VETO to his proposal (although
>>>> I don't have a binding vote, I think I have people who agree with me) if we
>>>> think we want to definitely expand the interpretation of VETO criteria in
>>>> the Apache Voting Process.
>>>>
>>>> You said it is up to the PMC member exercising the veto to use their
>>>> judgement, but definitely, it must not be used to force the community to
>>>> follow his proposal. The major argument here is, he can just VETO to any
>>>> proposal to retain the codebase as the way he prefers to, which I don't
>>>> believe is a correct usage of VETO.
>>>>
>>>> If we just revert the change of removal of config, this is "really"
>>>> neutral neither my proposal nor his proposal. Do we really want to do so?
>>>>
>>>> On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <ho...@gmail.com>
>>>> wrote:
>>>>
>>>>> First let me start with my key hope:
>>>>>
>>>>> We find a way to compromise and have the veto withdrawn rather than
>>>>> overridden.
>>>>>
>>>>> From what I understand of the change in question:
>>>>>
>>>>> So my understanding, and I may be over simplifying here but there are
>>>>> (at least) three technical paths forward (migration guide, legacy config
>>>>> with vendor string in it, non-vendor specific string legacy config), a PMC
>>>>> member vetoed one of them (named vendor legacy config) because he thought a
>>>>> different approach was better (migration guide) as they were worried that
>>>>> carrying that legacy config forward would encourage bad coding standards
>>>>> (eg we would add more vendor named config flags). To me that seems like a
>>>>> valid concern.
>>>>>
>>>>> My reasoning:
>>>>>
>>>>> Thinking back at other VETOs that I’ve been involved with in this
>>>>> project (DSV2, graceful decom, etc) this seems to meet the same bar. Hell
>>>>> we’ve had plenty of vetos that didn’t offer an alternative.
>>>>>
>>>>> My personal understanding of where the bar for “
>>>>> a technical justification showing why the change is bad” concern is
>>>>> pretty much “any not factually incorrect reasoning”, the text doesn’t have
>>>>> any particular “bar” for the level of “badness” and I think it’s up to the
>>>>> PMC member exercising the veto to use their judgement.
>>>>>
>>>>> In closing, I feel like the path we’re going down (overriding a veto)
>>>>> is not healthy for the project.
>>>>>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>> Pronouns: she/her
>>>>>
>>>>>
>>>>> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <
>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>
>>>>>> Holden, I believe you should already know "both" approaches are
>>>>>> "technically" correct. It's not about which one you have a preference for,
>>>>>> no, this VOTE is not intended to extend the debate.
>>>>>>
>>>>>> Again, what you are encouraged to do here is, not exposing your
>>>>>> preference of two approaches, but exposing your "technically valid" concern
>>>>>> of my approach, backed by Dongjoon's veto (most likely you want to quote
>>>>>> Dongjoon's post). This is very simple and I'm not sure you are doing
>>>>>> exactly what the VOTE requires.
>>>>>>
>>>>>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>>>>>> objection is technical then I think we need to respect the veto. There is a
>>>>>>> fundamental disagreement as to what the correct technical way to address
>>>>>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>>>>>> has vetoed  the legacy config option.
>>>>>>>
>>>>>>> I think I disagree with Mark on the assertion that the veto needs to
>>>>>>> have “substantial technical concern,” but rather a valid concern. I think
>>>>>>> in addition to the veto they’ve also gone above and beyond providing
>>>>>>> alternative ways to accomplish this.
>>>>>>>
>>>>>>> On a personal level:
>>>>>>>
>>>>>>> I am optimistic we can unblock the release but I think it’s
>>>>>>> important to err on the side of respecting the veto here in the interest of
>>>>>>> perceived fairness *especially* because of vendor aspects.
>>>>>>>
>>>>>>> To be clear I’ve worked at most of these companies (and many of the
>>>>>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>>>>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>>>>>> here and would ask that we show everyone understanding regardless of the
>>>>>>> outcome.
>>>>>>>
>>>>>>> Sending hugs and good vibes to y’all.
>>>>>>>
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>> Pronouns: she/her
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Given it’s the weekend maybe let’s give folks at least one full
>>>>>>>> work day.
>>>>>>>>
>>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>> Pronouns: she/her
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Quick administrative note: I don't see any reason why this vote
>>>>>>>>> should
>>>>>>>>> take a long time, so I expect to close the process and tally the
>>>>>>>>> votes
>>>>>>>>> in not much more than 48 hours.
>>>>>>>>>
>>>>>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <
>>>>>>>>> markhamstra@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > There has been enough discussion on this topic already, so I
>>>>>>>>> think
>>>>>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>>>>>> > justification for his veto of the "Retain migration logic ... in
>>>>>>>>> Spark
>>>>>>>>> > 4.0.x" proposal is in order. That technical justification has
>>>>>>>>> been
>>>>>>>>> > called into question, and the guidance at
>>>>>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it
>>>>>>>>> to the
>>>>>>>>> > PMC to determine whether the technical justification is  valid:
>>>>>>>>> "In
>>>>>>>>> > case of doubt, deciding whether a technical justification is
>>>>>>>>> valid is
>>>>>>>>> > up to the PMC." As such, only PMC votes will decide the outcome
>>>>>>>>> of
>>>>>>>>> > this vote. This is neither a vote on a code change itself not a
>>>>>>>>> vote
>>>>>>>>> > on whether a package is ready for release, so it a procedural
>>>>>>>>> vote on
>>>>>>>>> > whether the technical justification is valid. As such, the vote
>>>>>>>>> will
>>>>>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>>>>>> technical
>>>>>>>>> > justification is not valid and -1 votes hold that the technical
>>>>>>>>> > justification is valid.
>>>>>>>>> >
>>>>>>>>> > I would request that at least PMC members post more than just a
>>>>>>>>> naked
>>>>>>>>> > vote, but instead endeavor to give some reason why they have
>>>>>>>>> assessed
>>>>>>>>> > the technical justification as they have. I'll start:
>>>>>>>>> >
>>>>>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I
>>>>>>>>> must
>>>>>>>>> > confess to still not being entirely clear on what is his
>>>>>>>>> technical
>>>>>>>>> > justification for that veto. I see claims that including an
>>>>>>>>> admonition
>>>>>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>>>>>>> > required to maintain the integrity of already existing data
>>>>>>>>> streams,
>>>>>>>>> > and I see assertions about the maintenance burden that including
>>>>>>>>> the
>>>>>>>>> > migration logic would impose on future Spark versions, but I
>>>>>>>>> don't
>>>>>>>>> > think that I see any other technical objections. I do not
>>>>>>>>> believe that
>>>>>>>>> > the claimed technical justification is valid.
>>>>>>>>> >
>>>>>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>>>>>> > technical justification for the veto, the Apache Voting Process
>>>>>>>>> states
>>>>>>>>> > that: "To prevent vetoes from being used capriciously, the voter
>>>>>>>>> must
>>>>>>>>> > provide with the veto a technical justification showing why the
>>>>>>>>> change
>>>>>>>>> > is bad (opens a security exposure, negatively affects
>>>>>>>>> performance,
>>>>>>>>> > etc. ). A veto without a justification is invalid and has no
>>>>>>>>> weight."
>>>>>>>>> > This strongly implies that there must be something objectively
>>>>>>>>> wrong
>>>>>>>>> > with the proposed code change in that it causes significant harm
>>>>>>>>> in
>>>>>>>>> > the way of opening a security exposure, negatively affecting
>>>>>>>>> > performance, or presumably other significant user harms or
>>>>>>>>> perhaps
>>>>>>>>> > even developer burdens.
>>>>>>>>> >
>>>>>>>>> > The proposed addition of the migration logic to Spark 4.0.x does
>>>>>>>>> not
>>>>>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>>>>>> > streaming data, the change will have no effect. For streaming
>>>>>>>>> users
>>>>>>>>> > the change will be beneficial, not harmful.
>>>>>>>>> >
>>>>>>>>> > Neither do I find the claim of excessive, ongoing developer
>>>>>>>>> burden to
>>>>>>>>> > be persuasive. The changes are tiny and easily maintained -- in
>>>>>>>>> fact,
>>>>>>>>> > it wouldn't surprise me if no further changes to this migration
>>>>>>>>> logic
>>>>>>>>> > would be needed for a very long time.
>>>>>>>>> >
>>>>>>>>> > Some of what we are left with is just an expression of
>>>>>>>>> preference for
>>>>>>>>> > a technical alternative to the migration logic -- i.e. including
>>>>>>>>> in
>>>>>>>>> > the release notes an admonition to first upgrade to 3.5.5. But
>>>>>>>>> the
>>>>>>>>> > Apache Voting Process does not say that in the face of code
>>>>>>>>> > alternatives A and B, a qualified voter is justified in vetoing
>>>>>>>>> A if
>>>>>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>>>>>> > something more is needed to justify a veto, as I've already
>>>>>>>>> covered.
>>>>>>>>> > Thus I don't find Dongjoon's preference for the release notes
>>>>>>>>> option
>>>>>>>>> > to be adequate justification for the veto.
>>>>>>>>> >
>>>>>>>>> > The only remaining question I see is whether including
>>>>>>>>> "databricks" in
>>>>>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>>>>>> > expunged as soon as possible. I am not aware of any ASF policy
>>>>>>>>> that
>>>>>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>>>>>> > reason, even if that vendor has a product based on Apache code,
>>>>>>>>> even
>>>>>>>>> > if that vendor enjoys a uniquely influential position vis a vis
>>>>>>>>> some
>>>>>>>>> > Apache code or project. Certainly the PMC has a duty to see to
>>>>>>>>> it that
>>>>>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>>>>>> control
>>>>>>>>> > over Apache Spark outside of the established Apache process, but
>>>>>>>>> the
>>>>>>>>> > proposed migration code changes do not advantage Databricks -- if
>>>>>>>>> > anything they remove a minor avenue of influence, and simply
>>>>>>>>> need to
>>>>>>>>> > mention "databricks" once in order match and transform a
>>>>>>>>> configuration
>>>>>>>>> > into a vendor neutral equivalent. While not optimal, I can't
>>>>>>>>> find such
>>>>>>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>>>>>>> > non-technical policy concern -- certainly not offensive to the
>>>>>>>>> point
>>>>>>>>> > that it outweighs the user advantage of including the migration
>>>>>>>>> logic
>>>>>>>>> > in Spark 4.0.x.
>>>>>>>>> >
>>>>>>>>> > In summary, I do not find Dongjoon's given technical
>>>>>>>>> justification to
>>>>>>>>> > be valid relative to the Apache requirements for a veto of a code
>>>>>>>>> > change, so I must vote...
>>>>>>>>> >
>>>>>>>>> > +1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>
>>>>>>>>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Holden Karau <ho...@gmail.com>]

I'm delighted to see folks talking about a compromise. However, instead of
just asking Dongjoon to withdraw the VETO perhaps folks can suggest
alternatives that that might meet some of both parties goals?

On Sun, Mar 16, 2025 at 7:41 PM Wenchen Fan <cl...@gmail.com> wrote:

> I agree with Holden that withdrawing a veto is always better than
> overriding it: it's healthier for the community. Dongjoon, would you be
> willing to reconsider your veto given the current as-is state of the 4.0.0
> release (the breaking change will be reverted)?
>
> On Mon, Mar 17, 2025 at 10:36 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>> I've created the revert PR for branch-4.0:
>> https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
>> consensus but it's clear that this breaking change PR has failed to achieve
>> consensus.
>>
>> I hope we now have a clear foundation for discussing solutions. As it
>> stands, the misnamed configuration will be released in 4.0.0. I like
>> Jungtaek’s proposal to deprecate it, but the decision is up to the
>> community.
>>
>> On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> OK, let's be super honest.
>>>
>>> Again, I think you agree that *"both" proposals are "technically"
>>> correct (or one side can't have a strong theoretical evidence to counter
>>> the other side)*. So this naturally has a fate to have more supporters
>>> to get to the end. It's very easy for me to VETO to his proposal (although
>>> I don't have a binding vote, I think I have people who agree with me) if we
>>> think we want to definitely expand the interpretation of VETO criteria in
>>> the Apache Voting Process.
>>>
>>> You said it is up to the PMC member exercising the veto to use their
>>> judgement, but definitely, it must not be used to force the community to
>>> follow his proposal. The major argument here is, he can just VETO to any
>>> proposal to retain the codebase as the way he prefers to, which I don't
>>> believe is a correct usage of VETO.
>>>
>>> If we just revert the change of removal of config, this is "really"
>>> neutral neither my proposal nor his proposal. Do we really want to do so?
>>>
>>> On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> First let me start with my key hope:
>>>>
>>>> We find a way to compromise and have the veto withdrawn rather than
>>>> overridden.
>>>>
>>>> From what I understand of the change in question:
>>>>
>>>> So my understanding, and I may be over simplifying here but there are
>>>> (at least) three technical paths forward (migration guide, legacy config
>>>> with vendor string in it, non-vendor specific string legacy config), a PMC
>>>> member vetoed one of them (named vendor legacy config) because he thought a
>>>> different approach was better (migration guide) as they were worried that
>>>> carrying that legacy config forward would encourage bad coding standards
>>>> (eg we would add more vendor named config flags). To me that seems like a
>>>> valid concern.
>>>>
>>>> My reasoning:
>>>>
>>>> Thinking back at other VETOs that I’ve been involved with in this
>>>> project (DSV2, graceful decom, etc) this seems to meet the same bar. Hell
>>>> we’ve had plenty of vetos that didn’t offer an alternative.
>>>>
>>>> My personal understanding of where the bar for “
>>>> a technical justification showing why the change is bad” concern is
>>>> pretty much “any not factually incorrect reasoning”, the text doesn’t have
>>>> any particular “bar” for the level of “badness” and I think it’s up to the
>>>> PMC member exercising the veto to use their judgement.
>>>>
>>>> In closing, I feel like the path we’re going down (overriding a veto)
>>>> is not healthy for the project.
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>>
>>>> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>>
>>>>> Holden, I believe you should already know "both" approaches are
>>>>> "technically" correct. It's not about which one you have a preference for,
>>>>> no, this VOTE is not intended to extend the debate.
>>>>>
>>>>> Again, what you are encouraged to do here is, not exposing your
>>>>> preference of two approaches, but exposing your "technically valid" concern
>>>>> of my approach, backed by Dongjoon's veto (most likely you want to quote
>>>>> Dongjoon's post). This is very simple and I'm not sure you are doing
>>>>> exactly what the VOTE requires.
>>>>>
>>>>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>>>>> objection is technical then I think we need to respect the veto. There is a
>>>>>> fundamental disagreement as to what the correct technical way to address
>>>>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>>>>> has vetoed  the legacy config option.
>>>>>>
>>>>>> I think I disagree with Mark on the assertion that the veto needs to
>>>>>> have “substantial technical concern,” but rather a valid concern. I think
>>>>>> in addition to the veto they’ve also gone above and beyond providing
>>>>>> alternative ways to accomplish this.
>>>>>>
>>>>>> On a personal level:
>>>>>>
>>>>>> I am optimistic we can unblock the release but I think it’s important
>>>>>> to err on the side of respecting the veto here in the interest of perceived
>>>>>> fairness *especially* because of vendor aspects.
>>>>>>
>>>>>> To be clear I’ve worked at most of these companies (and many of the
>>>>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>>>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>>>>> here and would ask that we show everyone understanding regardless of the
>>>>>> outcome.
>>>>>>
>>>>>> Sending hugs and good vibes to y’all.
>>>>>>
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>> Pronouns: she/her
>>>>>>
>>>>>>
>>>>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Given it’s the weekend maybe let’s give folks at least one full work
>>>>>>> day.
>>>>>>>
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>> Pronouns: she/her
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Quick administrative note: I don't see any reason why this vote
>>>>>>>> should
>>>>>>>> take a long time, so I expect to close the process and tally the
>>>>>>>> votes
>>>>>>>> in not much more than 48 hours.
>>>>>>>>
>>>>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > There has been enough discussion on this topic already, so I think
>>>>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>>>>> > justification for his veto of the "Retain migration logic ... in
>>>>>>>> Spark
>>>>>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>>>>>> > called into question, and the guidance at
>>>>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it
>>>>>>>> to the
>>>>>>>> > PMC to determine whether the technical justification is  valid:
>>>>>>>> "In
>>>>>>>> > case of doubt, deciding whether a technical justification is
>>>>>>>> valid is
>>>>>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>>>>>> > this vote. This is neither a vote on a code change itself not a
>>>>>>>> vote
>>>>>>>> > on whether a package is ready for release, so it a procedural
>>>>>>>> vote on
>>>>>>>> > whether the technical justification is valid. As such, the vote
>>>>>>>> will
>>>>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>>>>> technical
>>>>>>>> > justification is not valid and -1 votes hold that the technical
>>>>>>>> > justification is valid.
>>>>>>>> >
>>>>>>>> > I would request that at least PMC members post more than just a
>>>>>>>> naked
>>>>>>>> > vote, but instead endeavor to give some reason why they have
>>>>>>>> assessed
>>>>>>>> > the technical justification as they have. I'll start:
>>>>>>>> >
>>>>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I
>>>>>>>> must
>>>>>>>> > confess to still not being entirely clear on what is his technical
>>>>>>>> > justification for that veto. I see claims that including an
>>>>>>>> admonition
>>>>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>>>>>> > required to maintain the integrity of already existing data
>>>>>>>> streams,
>>>>>>>> > and I see assertions about the maintenance burden that including
>>>>>>>> the
>>>>>>>> > migration logic would impose on future Spark versions, but I don't
>>>>>>>> > think that I see any other technical objections. I do not believe
>>>>>>>> that
>>>>>>>> > the claimed technical justification is valid.
>>>>>>>> >
>>>>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>>>>> > technical justification for the veto, the Apache Voting Process
>>>>>>>> states
>>>>>>>> > that: "To prevent vetoes from being used capriciously, the voter
>>>>>>>> must
>>>>>>>> > provide with the veto a technical justification showing why the
>>>>>>>> change
>>>>>>>> > is bad (opens a security exposure, negatively affects performance,
>>>>>>>> > etc. ). A veto without a justification is invalid and has no
>>>>>>>> weight."
>>>>>>>> > This strongly implies that there must be something objectively
>>>>>>>> wrong
>>>>>>>> > with the proposed code change in that it causes significant harm
>>>>>>>> in
>>>>>>>> > the way of opening a security exposure, negatively affecting
>>>>>>>> > performance, or presumably other significant user harms or perhaps
>>>>>>>> > even developer burdens.
>>>>>>>> >
>>>>>>>> > The proposed addition of the migration logic to Spark 4.0.x does
>>>>>>>> not
>>>>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>>>>> > streaming data, the change will have no effect. For streaming
>>>>>>>> users
>>>>>>>> > the change will be beneficial, not harmful.
>>>>>>>> >
>>>>>>>> > Neither do I find the claim of excessive, ongoing developer
>>>>>>>> burden to
>>>>>>>> > be persuasive. The changes are tiny and easily maintained -- in
>>>>>>>> fact,
>>>>>>>> > it wouldn't surprise me if no further changes to this migration
>>>>>>>> logic
>>>>>>>> > would be needed for a very long time.
>>>>>>>> >
>>>>>>>> > Some of what we are left with is just an expression of preference
>>>>>>>> for
>>>>>>>> > a technical alternative to the migration logic -- i.e. including
>>>>>>>> in
>>>>>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>>>>>> > Apache Voting Process does not say that in the face of code
>>>>>>>> > alternatives A and B, a qualified voter is justified in vetoing A
>>>>>>>> if
>>>>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>>>>> > something more is needed to justify a veto, as I've already
>>>>>>>> covered.
>>>>>>>> > Thus I don't find Dongjoon's preference for the release notes
>>>>>>>> option
>>>>>>>> > to be adequate justification for the veto.
>>>>>>>> >
>>>>>>>> > The only remaining question I see is whether including
>>>>>>>> "databricks" in
>>>>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>>>>> > expunged as soon as possible. I am not aware of any ASF policy
>>>>>>>> that
>>>>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>>>>> > reason, even if that vendor has a product based on Apache code,
>>>>>>>> even
>>>>>>>> > if that vendor enjoys a uniquely influential position vis a vis
>>>>>>>> some
>>>>>>>> > Apache code or project. Certainly the PMC has a duty to see to it
>>>>>>>> that
>>>>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>>>>> control
>>>>>>>> > over Apache Spark outside of the established Apache process, but
>>>>>>>> the
>>>>>>>> > proposed migration code changes do not advantage Databricks -- if
>>>>>>>> > anything they remove a minor avenue of influence, and simply need
>>>>>>>> to
>>>>>>>> > mention "databricks" once in order match and transform a
>>>>>>>> configuration
>>>>>>>> > into a vendor neutral equivalent. While not optimal, I can't find
>>>>>>>> such
>>>>>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>>>>>> > non-technical policy concern -- certainly not offensive to the
>>>>>>>> point
>>>>>>>> > that it outweighs the user advantage of including the migration
>>>>>>>> logic
>>>>>>>> > in Spark 4.0.x.
>>>>>>>> >
>>>>>>>> > In summary, I do not find Dongjoon's given technical
>>>>>>>> justification to
>>>>>>>> > be valid relative to the Apache requirements for a veto of a code
>>>>>>>> > change, so I must vote...
>>>>>>>> >
>>>>>>>> > +1
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>

-- 
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Wenchen Fan <cl...@gmail.com>]

I agree with Holden that withdrawing a veto is always better than
overriding it: it's healthier for the community. Dongjoon, would you be
willing to reconsider your veto given the current as-is state of the 4.0.0
release (the breaking change will be reverted)?

On Mon, Mar 17, 2025 at 10:36 AM Wenchen Fan <cl...@gmail.com> wrote:

> I've created the revert PR for branch-4.0:
> https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
> consensus but it's clear that this breaking change PR has failed to achieve
> consensus.
>
> I hope we now have a clear foundation for discussing solutions. As it
> stands, the misnamed configuration will be released in 4.0.0. I like
> Jungtaek’s proposal to deprecate it, but the decision is up to the
> community.
>
> On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim <
> kabhwan.opensource@gmail.com> wrote:
>
>> OK, let's be super honest.
>>
>> Again, I think you agree that *"both" proposals are "technically"
>> correct (or one side can't have a strong theoretical evidence to counter
>> the other side)*. So this naturally has a fate to have more supporters
>> to get to the end. It's very easy for me to VETO to his proposal (although
>> I don't have a binding vote, I think I have people who agree with me) if we
>> think we want to definitely expand the interpretation of VETO criteria in
>> the Apache Voting Process.
>>
>> You said it is up to the PMC member exercising the veto to use their
>> judgement, but definitely, it must not be used to force the community to
>> follow his proposal. The major argument here is, he can just VETO to any
>> proposal to retain the codebase as the way he prefers to, which I don't
>> believe is a correct usage of VETO.
>>
>> If we just revert the change of removal of config, this is "really"
>> neutral neither my proposal nor his proposal. Do we really want to do so?
>>
>> On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> First let me start with my key hope:
>>>
>>> We find a way to compromise and have the veto withdrawn rather than
>>> overridden.
>>>
>>> From what I understand of the change in question:
>>>
>>> So my understanding, and I may be over simplifying here but there are
>>> (at least) three technical paths forward (migration guide, legacy config
>>> with vendor string in it, non-vendor specific string legacy config), a PMC
>>> member vetoed one of them (named vendor legacy config) because he thought a
>>> different approach was better (migration guide) as they were worried that
>>> carrying that legacy config forward would encourage bad coding standards
>>> (eg we would add more vendor named config flags). To me that seems like a
>>> valid concern.
>>>
>>> My reasoning:
>>>
>>> Thinking back at other VETOs that I’ve been involved with in this
>>> project (DSV2, graceful decom, etc) this seems to meet the same bar. Hell
>>> we’ve had plenty of vetos that didn’t offer an alternative.
>>>
>>> My personal understanding of where the bar for “
>>> a technical justification showing why the change is bad” concern is
>>> pretty much “any not factually incorrect reasoning”, the text doesn’t have
>>> any particular “bar” for the level of “badness” and I think it’s up to the
>>> PMC member exercising the veto to use their judgement.
>>>
>>> In closing, I feel like the path we’re going down (overriding a veto) is
>>> not healthy for the project.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> Holden, I believe you should already know "both" approaches are
>>>> "technically" correct. It's not about which one you have a preference for,
>>>> no, this VOTE is not intended to extend the debate.
>>>>
>>>> Again, what you are encouraged to do here is, not exposing your
>>>> preference of two approaches, but exposing your "technically valid" concern
>>>> of my approach, backed by Dongjoon's veto (most likely you want to quote
>>>> Dongjoon's post). This is very simple and I'm not sure you are doing
>>>> exactly what the VOTE requires.
>>>>
>>>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>>>> wrote:
>>>>
>>>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>>>> objection is technical then I think we need to respect the veto. There is a
>>>>> fundamental disagreement as to what the correct technical way to address
>>>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>>>> has vetoed  the legacy config option.
>>>>>
>>>>> I think I disagree with Mark on the assertion that the veto needs to
>>>>> have “substantial technical concern,” but rather a valid concern. I think
>>>>> in addition to the veto they’ve also gone above and beyond providing
>>>>> alternative ways to accomplish this.
>>>>>
>>>>> On a personal level:
>>>>>
>>>>> I am optimistic we can unblock the release but I think it’s important
>>>>> to err on the side of respecting the veto here in the interest of perceived
>>>>> fairness *especially* because of vendor aspects.
>>>>>
>>>>> To be clear I’ve worked at most of these companies (and many of the
>>>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>>>> here and would ask that we show everyone understanding regardless of the
>>>>> outcome.
>>>>>
>>>>> Sending hugs and good vibes to y’all.
>>>>>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>> Pronouns: she/her
>>>>>
>>>>>
>>>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Given it’s the weekend maybe let’s give folks at least one full work
>>>>>> day.
>>>>>>
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>> Pronouns: she/her
>>>>>>
>>>>>>
>>>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Quick administrative note: I don't see any reason why this vote
>>>>>>> should
>>>>>>> take a long time, so I expect to close the process and tally the
>>>>>>> votes
>>>>>>> in not much more than 48 hours.
>>>>>>>
>>>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > There has been enough discussion on this topic already, so I think
>>>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>>>> > justification for his veto of the "Retain migration logic ... in
>>>>>>> Spark
>>>>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>>>>> > called into question, and the guidance at
>>>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to
>>>>>>> the
>>>>>>> > PMC to determine whether the technical justification is  valid: "In
>>>>>>> > case of doubt, deciding whether a technical justification is valid
>>>>>>> is
>>>>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>>>>> > this vote. This is neither a vote on a code change itself not a
>>>>>>> vote
>>>>>>> > on whether a package is ready for release, so it a procedural vote
>>>>>>> on
>>>>>>> > whether the technical justification is valid. As such, the vote
>>>>>>> will
>>>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>>>> technical
>>>>>>> > justification is not valid and -1 votes hold that the technical
>>>>>>> > justification is valid.
>>>>>>> >
>>>>>>> > I would request that at least PMC members post more than just a
>>>>>>> naked
>>>>>>> > vote, but instead endeavor to give some reason why they have
>>>>>>> assessed
>>>>>>> > the technical justification as they have. I'll start:
>>>>>>> >
>>>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>>>>>> > confess to still not being entirely clear on what is his technical
>>>>>>> > justification for that veto. I see claims that including an
>>>>>>> admonition
>>>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>>>>> > required to maintain the integrity of already existing data
>>>>>>> streams,
>>>>>>> > and I see assertions about the maintenance burden that including
>>>>>>> the
>>>>>>> > migration logic would impose on future Spark versions, but I don't
>>>>>>> > think that I see any other technical objections. I do not believe
>>>>>>> that
>>>>>>> > the claimed technical justification is valid.
>>>>>>> >
>>>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>>>> > technical justification for the veto, the Apache Voting Process
>>>>>>> states
>>>>>>> > that: "To prevent vetoes from being used capriciously, the voter
>>>>>>> must
>>>>>>> > provide with the veto a technical justification showing why the
>>>>>>> change
>>>>>>> > is bad (opens a security exposure, negatively affects performance,
>>>>>>> > etc. ). A veto without a justification is invalid and has no
>>>>>>> weight."
>>>>>>> > This strongly implies that there must be something objectively
>>>>>>> wrong
>>>>>>> > with the proposed code change in that it causes significant harm in
>>>>>>> > the way of opening a security exposure, negatively affecting
>>>>>>> > performance, or presumably other significant user harms or perhaps
>>>>>>> > even developer burdens.
>>>>>>> >
>>>>>>> > The proposed addition of the migration logic to Spark 4.0.x does
>>>>>>> not
>>>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>>>> > streaming data, the change will have no effect. For streaming users
>>>>>>> > the change will be beneficial, not harmful.
>>>>>>> >
>>>>>>> > Neither do I find the claim of excessive, ongoing developer burden
>>>>>>> to
>>>>>>> > be persuasive. The changes are tiny and easily maintained -- in
>>>>>>> fact,
>>>>>>> > it wouldn't surprise me if no further changes to this migration
>>>>>>> logic
>>>>>>> > would be needed for a very long time.
>>>>>>> >
>>>>>>> > Some of what we are left with is just an expression of preference
>>>>>>> for
>>>>>>> > a technical alternative to the migration logic -- i.e. including in
>>>>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>>>>> > Apache Voting Process does not say that in the face of code
>>>>>>> > alternatives A and B, a qualified voter is justified in vetoing A
>>>>>>> if
>>>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>>>> > something more is needed to justify a veto, as I've already
>>>>>>> covered.
>>>>>>> > Thus I don't find Dongjoon's preference for the release notes
>>>>>>> option
>>>>>>> > to be adequate justification for the veto.
>>>>>>> >
>>>>>>> > The only remaining question I see is whether including
>>>>>>> "databricks" in
>>>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>>>> > reason, even if that vendor has a product based on Apache code,
>>>>>>> even
>>>>>>> > if that vendor enjoys a uniquely influential position vis a vis
>>>>>>> some
>>>>>>> > Apache code or project. Certainly the PMC has a duty to see to it
>>>>>>> that
>>>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>>>> control
>>>>>>> > over Apache Spark outside of the established Apache process, but
>>>>>>> the
>>>>>>> > proposed migration code changes do not advantage Databricks -- if
>>>>>>> > anything they remove a minor avenue of influence, and simply need
>>>>>>> to
>>>>>>> > mention "databricks" once in order match and transform a
>>>>>>> configuration
>>>>>>> > into a vendor neutral equivalent. While not optimal, I can't find
>>>>>>> such
>>>>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>>>>> > non-technical policy concern -- certainly not offensive to the
>>>>>>> point
>>>>>>> > that it outweighs the user advantage of including the migration
>>>>>>> logic
>>>>>>> > in Spark 4.0.x.
>>>>>>> >
>>>>>>> > In summary, I do not find Dongjoon's given technical justification
>>>>>>> to
>>>>>>> > be valid relative to the Apache requirements for a veto of a code
>>>>>>> > change, so I must vote...
>>>>>>> >
>>>>>>> > +1
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

If we are really wanting to make a "correct" discussion going forward, I
believe the revert PR has to be merged. After that, either my proposal gets
not accepted, or he starts to DISCUSS and eventually reaches the VOTE
pass, or we just leave the config to be kept deprecated instead of removed.

We don't need to do this right now because this work is not necessary if
this VOTE has passed, but if this VOTE fails, I argue that the revert PR
must be merged, because the VOTE just means that he can just block my
proposal. It is never meant that he got consensus on his proposal. That
VOTE must happen separately, and during the time I want to see the codebase
to be "neutral".

On Mon, Mar 17, 2025 at 11:36 AM Wenchen Fan <cl...@gmail.com> wrote:

> I've created the revert PR for branch-4.0:
> https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
> consensus but it's clear that this breaking change PR has failed to achieve
> consensus.
>
> I hope we now have a clear foundation for discussing solutions. As it
> stands, the misnamed configuration will be released in 4.0.0. I like
> Jungtaek’s proposal to deprecate it, but the decision is up to the
> community.
>
> On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim <
> kabhwan.opensource@gmail.com> wrote:
>
>> OK, let's be super honest.
>>
>> Again, I think you agree that *"both" proposals are "technically"
>> correct (or one side can't have a strong theoretical evidence to counter
>> the other side)*. So this naturally has a fate to have more supporters
>> to get to the end. It's very easy for me to VETO to his proposal (although
>> I don't have a binding vote, I think I have people who agree with me) if we
>> think we want to definitely expand the interpretation of VETO criteria in
>> the Apache Voting Process.
>>
>> You said it is up to the PMC member exercising the veto to use their
>> judgement, but definitely, it must not be used to force the community to
>> follow his proposal. The major argument here is, he can just VETO to any
>> proposal to retain the codebase as the way he prefers to, which I don't
>> believe is a correct usage of VETO.
>>
>> If we just revert the change of removal of config, this is "really"
>> neutral neither my proposal nor his proposal. Do we really want to do so?
>>
>> On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> First let me start with my key hope:
>>>
>>> We find a way to compromise and have the veto withdrawn rather than
>>> overridden.
>>>
>>> From what I understand of the change in question:
>>>
>>> So my understanding, and I may be over simplifying here but there are
>>> (at least) three technical paths forward (migration guide, legacy config
>>> with vendor string in it, non-vendor specific string legacy config), a PMC
>>> member vetoed one of them (named vendor legacy config) because he thought a
>>> different approach was better (migration guide) as they were worried that
>>> carrying that legacy config forward would encourage bad coding standards
>>> (eg we would add more vendor named config flags). To me that seems like a
>>> valid concern.
>>>
>>> My reasoning:
>>>
>>> Thinking back at other VETOs that I’ve been involved with in this
>>> project (DSV2, graceful decom, etc) this seems to meet the same bar. Hell
>>> we’ve had plenty of vetos that didn’t offer an alternative.
>>>
>>> My personal understanding of where the bar for “
>>> a technical justification showing why the change is bad” concern is
>>> pretty much “any not factually incorrect reasoning”, the text doesn’t have
>>> any particular “bar” for the level of “badness” and I think it’s up to the
>>> PMC member exercising the veto to use their judgement.
>>>
>>> In closing, I feel like the path we’re going down (overriding a veto) is
>>> not healthy for the project.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> Holden, I believe you should already know "both" approaches are
>>>> "technically" correct. It's not about which one you have a preference for,
>>>> no, this VOTE is not intended to extend the debate.
>>>>
>>>> Again, what you are encouraged to do here is, not exposing your
>>>> preference of two approaches, but exposing your "technically valid" concern
>>>> of my approach, backed by Dongjoon's veto (most likely you want to quote
>>>> Dongjoon's post). This is very simple and I'm not sure you are doing
>>>> exactly what the VOTE requires.
>>>>
>>>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>>>> wrote:
>>>>
>>>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>>>> objection is technical then I think we need to respect the veto. There is a
>>>>> fundamental disagreement as to what the correct technical way to address
>>>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>>>> has vetoed  the legacy config option.
>>>>>
>>>>> I think I disagree with Mark on the assertion that the veto needs to
>>>>> have “substantial technical concern,” but rather a valid concern. I think
>>>>> in addition to the veto they’ve also gone above and beyond providing
>>>>> alternative ways to accomplish this.
>>>>>
>>>>> On a personal level:
>>>>>
>>>>> I am optimistic we can unblock the release but I think it’s important
>>>>> to err on the side of respecting the veto here in the interest of perceived
>>>>> fairness *especially* because of vendor aspects.
>>>>>
>>>>> To be clear I’ve worked at most of these companies (and many of the
>>>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>>>> here and would ask that we show everyone understanding regardless of the
>>>>> outcome.
>>>>>
>>>>> Sending hugs and good vibes to y’all.
>>>>>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>> Pronouns: she/her
>>>>>
>>>>>
>>>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Given it’s the weekend maybe let’s give folks at least one full work
>>>>>> day.
>>>>>>
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>> Pronouns: she/her
>>>>>>
>>>>>>
>>>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Quick administrative note: I don't see any reason why this vote
>>>>>>> should
>>>>>>> take a long time, so I expect to close the process and tally the
>>>>>>> votes
>>>>>>> in not much more than 48 hours.
>>>>>>>
>>>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > There has been enough discussion on this topic already, so I think
>>>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>>>> > justification for his veto of the "Retain migration logic ... in
>>>>>>> Spark
>>>>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>>>>> > called into question, and the guidance at
>>>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to
>>>>>>> the
>>>>>>> > PMC to determine whether the technical justification is  valid: "In
>>>>>>> > case of doubt, deciding whether a technical justification is valid
>>>>>>> is
>>>>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>>>>> > this vote. This is neither a vote on a code change itself not a
>>>>>>> vote
>>>>>>> > on whether a package is ready for release, so it a procedural vote
>>>>>>> on
>>>>>>> > whether the technical justification is valid. As such, the vote
>>>>>>> will
>>>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>>>> technical
>>>>>>> > justification is not valid and -1 votes hold that the technical
>>>>>>> > justification is valid.
>>>>>>> >
>>>>>>> > I would request that at least PMC members post more than just a
>>>>>>> naked
>>>>>>> > vote, but instead endeavor to give some reason why they have
>>>>>>> assessed
>>>>>>> > the technical justification as they have. I'll start:
>>>>>>> >
>>>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>>>>>> > confess to still not being entirely clear on what is his technical
>>>>>>> > justification for that veto. I see claims that including an
>>>>>>> admonition
>>>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>>>>> > required to maintain the integrity of already existing data
>>>>>>> streams,
>>>>>>> > and I see assertions about the maintenance burden that including
>>>>>>> the
>>>>>>> > migration logic would impose on future Spark versions, but I don't
>>>>>>> > think that I see any other technical objections. I do not believe
>>>>>>> that
>>>>>>> > the claimed technical justification is valid.
>>>>>>> >
>>>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>>>> > technical justification for the veto, the Apache Voting Process
>>>>>>> states
>>>>>>> > that: "To prevent vetoes from being used capriciously, the voter
>>>>>>> must
>>>>>>> > provide with the veto a technical justification showing why the
>>>>>>> change
>>>>>>> > is bad (opens a security exposure, negatively affects performance,
>>>>>>> > etc. ). A veto without a justification is invalid and has no
>>>>>>> weight."
>>>>>>> > This strongly implies that there must be something objectively
>>>>>>> wrong
>>>>>>> > with the proposed code change in that it causes significant harm in
>>>>>>> > the way of opening a security exposure, negatively affecting
>>>>>>> > performance, or presumably other significant user harms or perhaps
>>>>>>> > even developer burdens.
>>>>>>> >
>>>>>>> > The proposed addition of the migration logic to Spark 4.0.x does
>>>>>>> not
>>>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>>>> > streaming data, the change will have no effect. For streaming users
>>>>>>> > the change will be beneficial, not harmful.
>>>>>>> >
>>>>>>> > Neither do I find the claim of excessive, ongoing developer burden
>>>>>>> to
>>>>>>> > be persuasive. The changes are tiny and easily maintained -- in
>>>>>>> fact,
>>>>>>> > it wouldn't surprise me if no further changes to this migration
>>>>>>> logic
>>>>>>> > would be needed for a very long time.
>>>>>>> >
>>>>>>> > Some of what we are left with is just an expression of preference
>>>>>>> for
>>>>>>> > a technical alternative to the migration logic -- i.e. including in
>>>>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>>>>> > Apache Voting Process does not say that in the face of code
>>>>>>> > alternatives A and B, a qualified voter is justified in vetoing A
>>>>>>> if
>>>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>>>> > something more is needed to justify a veto, as I've already
>>>>>>> covered.
>>>>>>> > Thus I don't find Dongjoon's preference for the release notes
>>>>>>> option
>>>>>>> > to be adequate justification for the veto.
>>>>>>> >
>>>>>>> > The only remaining question I see is whether including
>>>>>>> "databricks" in
>>>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>>>> > reason, even if that vendor has a product based on Apache code,
>>>>>>> even
>>>>>>> > if that vendor enjoys a uniquely influential position vis a vis
>>>>>>> some
>>>>>>> > Apache code or project. Certainly the PMC has a duty to see to it
>>>>>>> that
>>>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>>>> control
>>>>>>> > over Apache Spark outside of the established Apache process, but
>>>>>>> the
>>>>>>> > proposed migration code changes do not advantage Databricks -- if
>>>>>>> > anything they remove a minor avenue of influence, and simply need
>>>>>>> to
>>>>>>> > mention "databricks" once in order match and transform a
>>>>>>> configuration
>>>>>>> > into a vendor neutral equivalent. While not optimal, I can't find
>>>>>>> such
>>>>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>>>>> > non-technical policy concern -- certainly not offensive to the
>>>>>>> point
>>>>>>> > that it outweighs the user advantage of including the migration
>>>>>>> logic
>>>>>>> > in Spark 4.0.x.
>>>>>>> >
>>>>>>> > In summary, I do not find Dongjoon's given technical justification
>>>>>>> to
>>>>>>> > be valid relative to the Apache requirements for a veto of a code
>>>>>>> > change, so I must vote...
>>>>>>> >
>>>>>>> > +1
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Wenchen Fan <cl...@gmail.com>]

I've created the revert PR for branch-4.0:
https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
consensus but it's clear that this breaking change PR has failed to achieve
consensus.

I hope we now have a clear foundation for discussing solutions. As it
stands, the misnamed configuration will be released in 4.0.0. I like
Jungtaek’s proposal to deprecate it, but the decision is up to the
community.

On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim <ka...@gmail.com>
wrote:

> OK, let's be super honest.
>
> Again, I think you agree that *"both" proposals are "technically" correct
> (or one side can't have a strong theoretical evidence to counter the other
> side)*. So this naturally has a fate to have more supporters to get to
> the end. It's very easy for me to VETO to his proposal (although I don't
> have a binding vote, I think I have people who agree with me) if we think
> we want to definitely expand the interpretation of VETO criteria in the
> Apache Voting Process.
>
> You said it is up to the PMC member exercising the veto to use their
> judgement, but definitely, it must not be used to force the community to
> follow his proposal. The major argument here is, he can just VETO to any
> proposal to retain the codebase as the way he prefers to, which I don't
> believe is a correct usage of VETO.
>
> If we just revert the change of removal of config, this is "really"
> neutral neither my proposal nor his proposal. Do we really want to do so?
>
> On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <ho...@gmail.com>
> wrote:
>
>> First let me start with my key hope:
>>
>> We find a way to compromise and have the veto withdrawn rather than
>> overridden.
>>
>> From what I understand of the change in question:
>>
>> So my understanding, and I may be over simplifying here but there are (at
>> least) three technical paths forward (migration guide, legacy config with
>> vendor string in it, non-vendor specific string legacy config), a PMC
>> member vetoed one of them (named vendor legacy config) because he thought a
>> different approach was better (migration guide) as they were worried that
>> carrying that legacy config forward would encourage bad coding standards
>> (eg we would add more vendor named config flags). To me that seems like a
>> valid concern.
>>
>> My reasoning:
>>
>> Thinking back at other VETOs that I’ve been involved with in this project
>> (DSV2, graceful decom, etc) this seems to meet the same bar. Hell we’ve had
>> plenty of vetos that didn’t offer an alternative.
>>
>> My personal understanding of where the bar for “
>> a technical justification showing why the change is bad” concern is
>> pretty much “any not factually incorrect reasoning”, the text doesn’t have
>> any particular “bar” for the level of “badness” and I think it’s up to the
>> PMC member exercising the veto to use their judgement.
>>
>> In closing, I feel like the path we’re going down (overriding a veto) is
>> not healthy for the project.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> Holden, I believe you should already know "both" approaches are
>>> "technically" correct. It's not about which one you have a preference for,
>>> no, this VOTE is not intended to extend the debate.
>>>
>>> Again, what you are encouraged to do here is, not exposing your
>>> preference of two approaches, but exposing your "technically valid" concern
>>> of my approach, backed by Dongjoon's veto (most likely you want to quote
>>> Dongjoon's post). This is very simple and I'm not sure you are doing
>>> exactly what the VOTE requires.
>>>
>>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>>> objection is technical then I think we need to respect the veto. There is a
>>>> fundamental disagreement as to what the correct technical way to address
>>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>>> has vetoed  the legacy config option.
>>>>
>>>> I think I disagree with Mark on the assertion that the veto needs to
>>>> have “substantial technical concern,” but rather a valid concern. I think
>>>> in addition to the veto they’ve also gone above and beyond providing
>>>> alternative ways to accomplish this.
>>>>
>>>> On a personal level:
>>>>
>>>> I am optimistic we can unblock the release but I think it’s important
>>>> to err on the side of respecting the veto here in the interest of perceived
>>>> fairness *especially* because of vendor aspects.
>>>>
>>>> To be clear I’ve worked at most of these companies (and many of the
>>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>>> here and would ask that we show everyone understanding regardless of the
>>>> outcome.
>>>>
>>>> Sending hugs and good vibes to y’all.
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>>
>>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>>>> wrote:
>>>>
>>>>> Given it’s the weekend maybe let’s give folks at least one full work
>>>>> day.
>>>>>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>> Pronouns: she/her
>>>>>
>>>>>
>>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Quick administrative note: I don't see any reason why this vote should
>>>>>> take a long time, so I expect to close the process and tally the votes
>>>>>> in not much more than 48 hours.
>>>>>>
>>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > There has been enough discussion on this topic already, so I think
>>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>>> > justification for his veto of the "Retain migration logic ... in
>>>>>> Spark
>>>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>>>> > called into question, and the guidance at
>>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to
>>>>>> the
>>>>>> > PMC to determine whether the technical justification is  valid: "In
>>>>>> > case of doubt, deciding whether a technical justification is valid
>>>>>> is
>>>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>>>> > this vote. This is neither a vote on a code change itself not a vote
>>>>>> > on whether a package is ready for release, so it a procedural vote
>>>>>> on
>>>>>> > whether the technical justification is valid. As such, the vote will
>>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>>> technical
>>>>>> > justification is not valid and -1 votes hold that the technical
>>>>>> > justification is valid.
>>>>>> >
>>>>>> > I would request that at least PMC members post more than just a
>>>>>> naked
>>>>>> > vote, but instead endeavor to give some reason why they have
>>>>>> assessed
>>>>>> > the technical justification as they have. I'll start:
>>>>>> >
>>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>>>>> > confess to still not being entirely clear on what is his technical
>>>>>> > justification for that veto. I see claims that including an
>>>>>> admonition
>>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>>>> > required to maintain the integrity of already existing data streams,
>>>>>> > and I see assertions about the maintenance burden that including the
>>>>>> > migration logic would impose on future Spark versions, but I don't
>>>>>> > think that I see any other technical objections. I do not believe
>>>>>> that
>>>>>> > the claimed technical justification is valid.
>>>>>> >
>>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>>> > technical justification for the veto, the Apache Voting Process
>>>>>> states
>>>>>> > that: "To prevent vetoes from being used capriciously, the voter
>>>>>> must
>>>>>> > provide with the veto a technical justification showing why the
>>>>>> change
>>>>>> > is bad (opens a security exposure, negatively affects performance,
>>>>>> > etc. ). A veto without a justification is invalid and has no
>>>>>> weight."
>>>>>> > This strongly implies that there must be something objectively wrong
>>>>>> > with the proposed code change in that it causes significant harm in
>>>>>> > the way of opening a security exposure, negatively affecting
>>>>>> > performance, or presumably other significant user harms or perhaps
>>>>>> > even developer burdens.
>>>>>> >
>>>>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>>> > streaming data, the change will have no effect. For streaming users
>>>>>> > the change will be beneficial, not harmful.
>>>>>> >
>>>>>> > Neither do I find the claim of excessive, ongoing developer burden
>>>>>> to
>>>>>> > be persuasive. The changes are tiny and easily maintained -- in
>>>>>> fact,
>>>>>> > it wouldn't surprise me if no further changes to this migration
>>>>>> logic
>>>>>> > would be needed for a very long time.
>>>>>> >
>>>>>> > Some of what we are left with is just an expression of preference
>>>>>> for
>>>>>> > a technical alternative to the migration logic -- i.e. including in
>>>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>>>> > Apache Voting Process does not say that in the face of code
>>>>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>>> > something more is needed to justify a veto, as I've already covered.
>>>>>> > Thus I don't find Dongjoon's preference for the release notes option
>>>>>> > to be adequate justification for the veto.
>>>>>> >
>>>>>> > The only remaining question I see is whether including "databricks"
>>>>>> in
>>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>>> > reason, even if that vendor has a product based on Apache code, even
>>>>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>>>>> > Apache code or project. Certainly the PMC has a duty to see to it
>>>>>> that
>>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>>> control
>>>>>> > over Apache Spark outside of the established Apache process, but the
>>>>>> > proposed migration code changes do not advantage Databricks -- if
>>>>>> > anything they remove a minor avenue of influence, and simply need to
>>>>>> > mention "databricks" once in order match and transform a
>>>>>> configuration
>>>>>> > into a vendor neutral equivalent. While not optimal, I can't find
>>>>>> such
>>>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>>>> > non-technical policy concern -- certainly not offensive to the point
>>>>>> > that it outweighs the user advantage of including the migration
>>>>>> logic
>>>>>> > in Spark 4.0.x.
>>>>>> >
>>>>>> > In summary, I do not find Dongjoon's given technical justification
>>>>>> to
>>>>>> > be valid relative to the Apache requirements for a veto of a code
>>>>>> > change, so I must vote...
>>>>>> >
>>>>>> > +1
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

OK, let's be super honest.

Again, I think you agree that *"both" proposals are "technically" correct
(or one side can't have a strong theoretical evidence to counter the other
side)*. So this naturally has a fate to have more supporters to get to the
end. It's very easy for me to VETO to his proposal (although I don't have a
binding vote, I think I have people who agree with me) if we think we want
to definitely expand the interpretation of VETO criteria in the Apache
Voting Process.

You said it is up to the PMC member exercising the veto to use their
judgement, but definitely, it must not be used to force the community to
follow his proposal. The major argument here is, he can just VETO to any
proposal to retain the codebase as the way he prefers to, which I don't
believe is a correct usage of VETO.

If we just revert the change of removal of config, this is "really" neutral
neither my proposal nor his proposal. Do we really want to do so?

On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <ho...@gmail.com>
wrote:

> First let me start with my key hope:
>
> We find a way to compromise and have the veto withdrawn rather than
> overridden.
>
> From what I understand of the change in question:
>
> So my understanding, and I may be over simplifying here but there are (at
> least) three technical paths forward (migration guide, legacy config with
> vendor string in it, non-vendor specific string legacy config), a PMC
> member vetoed one of them (named vendor legacy config) because he thought a
> different approach was better (migration guide) as they were worried that
> carrying that legacy config forward would encourage bad coding standards
> (eg we would add more vendor named config flags). To me that seems like a
> valid concern.
>
> My reasoning:
>
> Thinking back at other VETOs that I’ve been involved with in this project
> (DSV2, graceful decom, etc) this seems to meet the same bar. Hell we’ve had
> plenty of vetos that didn’t offer an alternative.
>
> My personal understanding of where the bar for “
> a technical justification showing why the change is bad” concern is
> pretty much “any not factually incorrect reasoning”, the text doesn’t have
> any particular “bar” for the level of “badness” and I think it’s up to the
> PMC member exercising the veto to use their judgement.
>
> In closing, I feel like the path we’re going down (overriding a veto) is
> not healthy for the project.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Holden, I believe you should already know "both" approaches are
>> "technically" correct. It's not about which one you have a preference for,
>> no, this VOTE is not intended to extend the debate.
>>
>> Again, what you are encouraged to do here is, not exposing your
>> preference of two approaches, but exposing your "technically valid" concern
>> of my approach, backed by Dongjoon's veto (most likely you want to quote
>> Dongjoon's post). This is very simple and I'm not sure you are doing
>> exactly what the VOTE requires.
>>
>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>> objection is technical then I think we need to respect the veto. There is a
>>> fundamental disagreement as to what the correct technical way to address
>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>> has vetoed  the legacy config option.
>>>
>>> I think I disagree with Mark on the assertion that the veto needs to
>>> have “substantial technical concern,” but rather a valid concern. I think
>>> in addition to the veto they’ve also gone above and beyond providing
>>> alternative ways to accomplish this.
>>>
>>> On a personal level:
>>>
>>> I am optimistic we can unblock the release but I think it’s important to
>>> err on the side of respecting the veto here in the interest of perceived
>>> fairness *especially* because of vendor aspects.
>>>
>>> To be clear I’ve worked at most of these companies (and many of the
>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>> here and would ask that we show everyone understanding regardless of the
>>> outcome.
>>>
>>> Sending hugs and good vibes to y’all.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> Given it’s the weekend maybe let’s give folks at least one full work
>>>> day.
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>>
>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Quick administrative note: I don't see any reason why this vote should
>>>>> take a long time, so I expect to close the process and tally the votes
>>>>> in not much more than 48 hours.
>>>>>
>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > There has been enough discussion on this topic already, so I think
>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>> > justification for his veto of the "Retain migration logic ... in
>>>>> Spark
>>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>>> > called into question, and the guidance at
>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to
>>>>> the
>>>>> > PMC to determine whether the technical justification is  valid: "In
>>>>> > case of doubt, deciding whether a technical justification is valid is
>>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>>> > this vote. This is neither a vote on a code change itself not a vote
>>>>> > on whether a package is ready for release, so it a procedural vote on
>>>>> > whether the technical justification is valid. As such, the vote will
>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>> technical
>>>>> > justification is not valid and -1 votes hold that the technical
>>>>> > justification is valid.
>>>>> >
>>>>> > I would request that at least PMC members post more than just a naked
>>>>> > vote, but instead endeavor to give some reason why they have assessed
>>>>> > the technical justification as they have. I'll start:
>>>>> >
>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>>>> > confess to still not being entirely clear on what is his technical
>>>>> > justification for that veto. I see claims that including an
>>>>> admonition
>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>>> > required to maintain the integrity of already existing data streams,
>>>>> > and I see assertions about the maintenance burden that including the
>>>>> > migration logic would impose on future Spark versions, but I don't
>>>>> > think that I see any other technical objections. I do not believe
>>>>> that
>>>>> > the claimed technical justification is valid.
>>>>> >
>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>> > technical justification for the veto, the Apache Voting Process
>>>>> states
>>>>> > that: "To prevent vetoes from being used capriciously, the voter must
>>>>> > provide with the veto a technical justification showing why the
>>>>> change
>>>>> > is bad (opens a security exposure, negatively affects performance,
>>>>> > etc. ). A veto without a justification is invalid and has no weight."
>>>>> > This strongly implies that there must be something objectively wrong
>>>>> > with the proposed code change in that it causes significant harm in
>>>>> > the way of opening a security exposure, negatively affecting
>>>>> > performance, or presumably other significant user harms or perhaps
>>>>> > even developer burdens.
>>>>> >
>>>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>> > streaming data, the change will have no effect. For streaming users
>>>>> > the change will be beneficial, not harmful.
>>>>> >
>>>>> > Neither do I find the claim of excessive, ongoing developer burden to
>>>>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>>>>> > it wouldn't surprise me if no further changes to this migration logic
>>>>> > would be needed for a very long time.
>>>>> >
>>>>> > Some of what we are left with is just an expression of preference for
>>>>> > a technical alternative to the migration logic -- i.e. including in
>>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>>> > Apache Voting Process does not say that in the face of code
>>>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>> > something more is needed to justify a veto, as I've already covered.
>>>>> > Thus I don't find Dongjoon's preference for the release notes option
>>>>> > to be adequate justification for the veto.
>>>>> >
>>>>> > The only remaining question I see is whether including "databricks"
>>>>> in
>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>> > reason, even if that vendor has a product based on Apache code, even
>>>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>>>> > Apache code or project. Certainly the PMC has a duty to see to it
>>>>> that
>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>> control
>>>>> > over Apache Spark outside of the established Apache process, but the
>>>>> > proposed migration code changes do not advantage Databricks -- if
>>>>> > anything they remove a minor avenue of influence, and simply need to
>>>>> > mention "databricks" once in order match and transform a
>>>>> configuration
>>>>> > into a vendor neutral equivalent. While not optimal, I can't find
>>>>> such
>>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>>> > non-technical policy concern -- certainly not offensive to the point
>>>>> > that it outweighs the user advantage of including the migration logic
>>>>> > in Spark 4.0.x.
>>>>> >
>>>>> > In summary, I do not find Dongjoon's given technical justification to
>>>>> > be valid relative to the Apache requirements for a veto of a code
>>>>> > change, so I must vote...
>>>>> >
>>>>> > +1
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Holden Karau <ho...@gmail.com>]

First let me start with my key hope:

We find a way to compromise and have the veto withdrawn rather than
overridden.

From what I understand of the change in question:

So my understanding, and I may be over simplifying here but there are (at
least) three technical paths forward (migration guide, legacy config with
vendor string in it, non-vendor specific string legacy config), a PMC
member vetoed one of them (named vendor legacy config) because he thought a
different approach was better (migration guide) as they were worried that
carrying that legacy config forward would encourage bad coding standards
(eg we would add more vendor named config flags). To me that seems like a
valid concern.

My reasoning:

Thinking back at other VETOs that I’ve been involved with in this project
(DSV2, graceful decom, etc) this seems to meet the same bar. Hell we’ve had
plenty of vetos that didn’t offer an alternative.

My personal understanding of where the bar for “
a technical justification showing why the change is bad” concern is pretty
much “any not factually incorrect reasoning”, the text doesn’t have any
particular “bar” for the level of “badness” and I think it’s up to the PMC
member exercising the veto to use their judgement.

In closing, I feel like the path we’re going down (overriding a veto) is
not healthy for the project.

Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her


On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Holden, I believe you should already know "both" approaches are
> "technically" correct. It's not about which one you have a preference for,
> no, this VOTE is not intended to extend the debate.
>
> Again, what you are encouraged to do here is, not exposing your preference
> of two approaches, but exposing your "technically valid" concern of my
> approach, backed by Dongjoon's veto (most likely you want to quote
> Dongjoon's post). This is very simple and I'm not sure you are doing
> exactly what the VOTE requires.
>
> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
> wrote:
>
>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>> objection is technical then I think we need to respect the veto. There is a
>> fundamental disagreement as to what the correct technical way to address
>> this problem is (removal + documentation vs legacy config) and a PMC member
>> has vetoed  the legacy config option.
>>
>> I think I disagree with Mark on the assertion that the veto needs to have
>> “substantial technical concern,” but rather a valid concern. I think in
>> addition to the veto they’ve also gone above and beyond providing
>> alternative ways to accomplish this.
>>
>> On a personal level:
>>
>> I am optimistic we can unblock the release but I think it’s important to
>> err on the side of respecting the veto here in the interest of perceived
>> fairness *especially* because of vendor aspects.
>>
>> To be clear I’ve worked at most of these companies (and many of the
>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>> here and would ask that we show everyone understanding regardless of the
>> outcome.
>>
>> Sending hugs and good vibes to y’all.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> Given it’s the weekend maybe let’s give folks at least one full work day.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>> wrote:
>>>
>>>> Quick administrative note: I don't see any reason why this vote should
>>>> take a long time, so I expect to close the process and tally the votes
>>>> in not much more than 48 hours.
>>>>
>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>> >
>>>> > There has been enough discussion on this topic already, so I think
>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>> > justification for his veto of the "Retain migration logic ... in Spark
>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>> > called into question, and the guidance at
>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>>>> > PMC to determine whether the technical justification is  valid: "In
>>>> > case of doubt, deciding whether a technical justification is valid is
>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>> > this vote. This is neither a vote on a code change itself not a vote
>>>> > on whether a package is ready for release, so it a procedural vote on
>>>> > whether the technical justification is valid. As such, the vote will
>>>> > be decided by a simple majority where +1 votes hold that the technical
>>>> > justification is not valid and -1 votes hold that the technical
>>>> > justification is valid.
>>>> >
>>>> > I would request that at least PMC members post more than just a naked
>>>> > vote, but instead endeavor to give some reason why they have assessed
>>>> > the technical justification as they have. I'll start:
>>>> >
>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>>> > confess to still not being entirely clear on what is his technical
>>>> > justification for that veto. I see claims that including an admonition
>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>> > required to maintain the integrity of already existing data streams,
>>>> > and I see assertions about the maintenance burden that including the
>>>> > migration logic would impose on future Spark versions, but I don't
>>>> > think that I see any other technical objections. I do not believe that
>>>> > the claimed technical justification is valid.
>>>> >
>>>> > In requiring that a veto of a code change be accompanied by a
>>>> > technical justification for the veto, the Apache Voting Process states
>>>> > that: "To prevent vetoes from being used capriciously, the voter must
>>>> > provide with the veto a technical justification showing why the change
>>>> > is bad (opens a security exposure, negatively affects performance,
>>>> > etc. ). A veto without a justification is invalid and has no weight."
>>>> > This strongly implies that there must be something objectively wrong
>>>> > with the proposed code change in that it causes significant harm in
>>>> > the way of opening a security exposure, negatively affecting
>>>> > performance, or presumably other significant user harms or perhaps
>>>> > even developer burdens.
>>>> >
>>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>>> > cause any harm to Spark's users. For many users, those not using
>>>> > streaming data, the change will have no effect. For streaming users
>>>> > the change will be beneficial, not harmful.
>>>> >
>>>> > Neither do I find the claim of excessive, ongoing developer burden to
>>>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>>>> > it wouldn't surprise me if no further changes to this migration logic
>>>> > would be needed for a very long time.
>>>> >
>>>> > Some of what we are left with is just an expression of preference for
>>>> > a technical alternative to the migration logic -- i.e. including in
>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>> > Apache Voting Process does not say that in the face of code
>>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>> > something more is needed to justify a veto, as I've already covered.
>>>> > Thus I don't find Dongjoon's preference for the release notes option
>>>> > to be adequate justification for the veto.
>>>> >
>>>> > The only remaining question I see is whether including "databricks" in
>>>> > the Apache Code is ever allowed or if any such instance must be
>>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>> > reason, even if that vendor has a product based on Apache code, even
>>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>>> > Apache code or project. Certainly the PMC has a duty to see to it that
>>>> > neither Databricks nor any other vendor exercises influence or control
>>>> > over Apache Spark outside of the established Apache process, but the
>>>> > proposed migration code changes do not advantage Databricks -- if
>>>> > anything they remove a minor avenue of influence, and simply need to
>>>> > mention "databricks" once in order match and transform a configuration
>>>> > into a vendor neutral equivalent. While not optimal, I can't find such
>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>> > non-technical policy concern -- certainly not offensive to the point
>>>> > that it outweighs the user advantage of including the migration logic
>>>> > in Spark 4.0.x.
>>>> >
>>>> > In summary, I do not find Dongjoon's given technical justification to
>>>> > be valid relative to the Apache requirements for a veto of a code
>>>> > change, so I must vote...
>>>> >
>>>> > +1
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Holden, I believe you should already know "both" approaches are
"technically" correct. It's not about which one you have a preference for,
no, this VOTE is not intended to extend the debate.

Again, what you are encouraged to do here is, not exposing your preference
of two approaches, but exposing your "technically valid" concern of my
approach, backed by Dongjoon's veto (most likely you want to quote
Dongjoon's post). This is very simple and I'm not sure you are doing
exactly what the VOTE requires.

On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com> wrote:

> -1 (binding) — to me it doesn’t matter that the cost is low if the
> objection is technical then I think we need to respect the veto. There is a
> fundamental disagreement as to what the correct technical way to address
> this problem is (removal + documentation vs legacy config) and a PMC member
> has vetoed  the legacy config option.
>
> I think I disagree with Mark on the assertion that the veto needs to have
> “substantial technical concern,” but rather a valid concern. I think in
> addition to the veto they’ve also gone above and beyond providing
> alternative ways to accomplish this.
>
> On a personal level:
>
> I am optimistic we can unblock the release but I think it’s important to
> err on the side of respecting the veto here in the interest of perceived
> fairness *especially* because of vendor aspects.
>
> To be clear I’ve worked at most of these companies (and many of the
> people) and I’m not ascribing malice to anyone in this, I think mistakes
> happen (god knows I’ve had a fair share). I think we’re all doing our best
> here and would ask that we show everyone understanding regardless of the
> outcome.
>
> Sending hugs and good vibes to y’all.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
> wrote:
>
>> Given it’s the weekend maybe let’s give folks at least one full work day.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Quick administrative note: I don't see any reason why this vote should
>>> take a long time, so I expect to close the process and tally the votes
>>> in not much more than 48 hours.
>>>
>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>> wrote:
>>> >
>>> > There has been enough discussion on this topic already, so I think
>>> > that an immediate vote on the validity of Dongjoon's technical
>>> > justification for his veto of the "Retain migration logic ... in Spark
>>> > 4.0.x" proposal is in order. That technical justification has been
>>> > called into question, and the guidance at
>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>>> > PMC to determine whether the technical justification is  valid: "In
>>> > case of doubt, deciding whether a technical justification is valid is
>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>> > this vote. This is neither a vote on a code change itself not a vote
>>> > on whether a package is ready for release, so it a procedural vote on
>>> > whether the technical justification is valid. As such, the vote will
>>> > be decided by a simple majority where +1 votes hold that the technical
>>> > justification is not valid and -1 votes hold that the technical
>>> > justification is valid.
>>> >
>>> > I would request that at least PMC members post more than just a naked
>>> > vote, but instead endeavor to give some reason why they have assessed
>>> > the technical justification as they have. I'll start:
>>> >
>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>> > confess to still not being entirely clear on what is his technical
>>> > justification for that veto. I see claims that including an admonition
>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>> > required to maintain the integrity of already existing data streams,
>>> > and I see assertions about the maintenance burden that including the
>>> > migration logic would impose on future Spark versions, but I don't
>>> > think that I see any other technical objections. I do not believe that
>>> > the claimed technical justification is valid.
>>> >
>>> > In requiring that a veto of a code change be accompanied by a
>>> > technical justification for the veto, the Apache Voting Process states
>>> > that: "To prevent vetoes from being used capriciously, the voter must
>>> > provide with the veto a technical justification showing why the change
>>> > is bad (opens a security exposure, negatively affects performance,
>>> > etc. ). A veto without a justification is invalid and has no weight."
>>> > This strongly implies that there must be something objectively wrong
>>> > with the proposed code change in that it causes significant harm in
>>> > the way of opening a security exposure, negatively affecting
>>> > performance, or presumably other significant user harms or perhaps
>>> > even developer burdens.
>>> >
>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>> > cause any harm to Spark's users. For many users, those not using
>>> > streaming data, the change will have no effect. For streaming users
>>> > the change will be beneficial, not harmful.
>>> >
>>> > Neither do I find the claim of excessive, ongoing developer burden to
>>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>>> > it wouldn't surprise me if no further changes to this migration logic
>>> > would be needed for a very long time.
>>> >
>>> > Some of what we are left with is just an expression of preference for
>>> > a technical alternative to the migration logic -- i.e. including in
>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>> > Apache Voting Process does not say that in the face of code
>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>> > they prefer B. Instead, the Voting Process strongly implies that
>>> > something more is needed to justify a veto, as I've already covered.
>>> > Thus I don't find Dongjoon's preference for the release notes option
>>> > to be adequate justification for the veto.
>>> >
>>> > The only remaining question I see is whether including "databricks" in
>>> > the Apache Code is ever allowed or if any such instance must be
>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>> > strictly forbids the mention of a vendor in Apache code for any
>>> > reason, even if that vendor has a product based on Apache code, even
>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>> > Apache code or project. Certainly the PMC has a duty to see to it that
>>> > neither Databricks nor any other vendor exercises influence or control
>>> > over Apache Spark outside of the established Apache process, but the
>>> > proposed migration code changes do not advantage Databricks -- if
>>> > anything they remove a minor avenue of influence, and simply need to
>>> > mention "databricks" once in order match and transform a configuration
>>> > into a vendor neutral equivalent. While not optimal, I can't find such
>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>> > non-technical policy concern -- certainly not offensive to the point
>>> > that it outweighs the user advantage of including the migration logic
>>> > in Spark 4.0.x.
>>> >
>>> > In summary, I do not find Dongjoon's given technical justification to
>>> > be valid relative to the Apache requirements for a veto of a code
>>> > change, so I must vote...
>>> >
>>> > +1
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

It's OK, I just wanted to let him give the link whenever he makes the
argument. It is NOT important that he brings up a new reasoning at this
point. It must have been there in VOTE.

Also, I think we all agree that this is NOT a place of debate about having
vendor names in the codebase. I think I gave plenty of time from DISCUSSION
and VOTE to let PMC members chime in about their concerns.

Let me remind - this VOTE is, to judge, of Dongjoon's VETO. Again, Mark
clarified the criteria of VETO from quoting ASF page:

>
> In requiring that a veto of a code change be accompanied by a
> technical justification for the veto, the Apache Voting Process states
> that: "To prevent vetoes from being used capriciously, the voter must
> provide with the veto a technical justification showing why the change
> is bad (opens a security exposure, negatively affects performance,
> etc. ). A veto without a justification is invalid and has no weight."
> This strongly implies that there must be something objectively wrong
> with the proposed code change in that it causes significant harm in
> the way of opening a security exposure, negatively affecting
> performance, or presumably other significant user harms or perhaps
> even developer burdens.


Anyone speaking for the judge, the above must be the central reasoning to
do so. If the claim is sidetracked from this criteria of VETO, this just
expands the previous VOTE which we intend to initiate this VOTE to "avoid"
it.

On Mon, Mar 17, 2025 at 7:40 AM Holden Karau <ho...@gmail.com> wrote:

> I believe he intended to VETO this change for Spark 4, of course if
> Dongjoon did not (or no longer intends to) then this VOTE becomes moot. I
> think bringing up 3.5.5 confuses the issue -- this vote thread is very
> clearly about VETOing the change for Spark 4.
>
> I think that accusing each-other of acting in bad faith is unproductive to
> resolving this dispute.
>
> On Sun, Mar 16, 2025 at 3:14 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Holden and Dongjoon,
>>
>> Let me make this vote super simple. I never got the answer from Dongjoon
>> about this question. This is super important because if he's casting veto
>> "to block", it is a strong indication that he was intended to play with me,
>> which I am seriously considering escalating the problem (If this is true,
>> it's no longer just a justification of vote, but someone's
>> behavioral issue).
>>
>> https://github.com/apache/spark/pull/49983
>>
>> I might be missing another timeline, but, if you follow the conversation
>> here, there are some facts:
>>
>> 1. Dongjoon "knew" we were never decided about the direction of Spark
>> 4.0.0 behavior. (link
>> <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>)
>> 2. Dongjoon "agreed" my proposal is technically correct. (link
>> <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>)
>> 3. Dongjoon "agreed" to hear from the community about discussing my
>> proposal. (link
>> <https://github.com/apache/spark/pull/49983#issuecomment-2696064545>)
>>
>> Worth clarifying, 3 happened after we discussed the removal of "config".
>> Dongjoon continuously mixed up the fact - while we were in agreement of
>> removal of config, removal of migration logic was definitely left to open
>> question. Let me give the VOTE Dongjoon drove and made it pass.
>>
>> https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5
>>
>> This was totally about 3.5.5. If Dongjoon thinks this simply applies to
>> Spark 4.0.0+, it's not, no?
>>
>> Also, let's revisit the discussion we were discussing about removal of
>> config.
>>
>> https://lists.apache.org/thread/qxqzt7wbtdyxp17d7s1rxhnrswdccsgb
>>
>> Dongjoon clearly stated that we only make a consensus about Spark 3.5.5,
>> and we can continue discussion about the proper behavior in Spark 4.0.0.
>> That is the rationale I drove my own discussion. I can be corrected, but
>> there is NO discussion/vote w.r.t this topic AFAIK.
>>
>> Dongjoon, now it's your time to prove there is a valid reason to change
>> your mind during this time frame. If the above are all true, you are
>> already indicating that you can never cast a veto. (Or show me the evidence
>> of how you change your mind for which reason.) If any of the above are
>> something you intended to not tell the truth, I am really not sure your
>> comment will be truthful I can follow. Especially, if you did not tell the
>> truth from 3, e.g. you let me go and discuss while you were intended to
>> block me in any phase, this is a strong indication that you intend to play
>> with me and the community (or even ASF) has to know that.
>>
>> Do not evade the root question.
>>
>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>>> objection is technical then I think we need to respect the veto. There is a
>>> fundamental disagreement as to what the correct technical way to address
>>> this problem is (removal + documentation vs legacy config) and a PMC member
>>> has vetoed  the legacy config option.
>>>
>>> I think I disagree with Mark on the assertion that the veto needs to
>>> have “substantial technical concern,” but rather a valid concern. I think
>>> in addition to the veto they’ve also gone above and beyond providing
>>> alternative ways to accomplish this.
>>>
>>> On a personal level:
>>>
>>> I am optimistic we can unblock the release but I think it’s important to
>>> err on the side of respecting the veto here in the interest of perceived
>>> fairness *especially* because of vendor aspects.
>>>
>>> To be clear I’ve worked at most of these companies (and many of the
>>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>>> here and would ask that we show everyone understanding regardless of the
>>> outcome.
>>>
>>> Sending hugs and good vibes to y’all.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> Given it’s the weekend maybe let’s give folks at least one full work
>>>> day.
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>>
>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Quick administrative note: I don't see any reason why this vote should
>>>>> take a long time, so I expect to close the process and tally the votes
>>>>> in not much more than 48 hours.
>>>>>
>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > There has been enough discussion on this topic already, so I think
>>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>>> > justification for his veto of the "Retain migration logic ... in
>>>>> Spark
>>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>>> > called into question, and the guidance at
>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to
>>>>> the
>>>>> > PMC to determine whether the technical justification is  valid: "In
>>>>> > case of doubt, deciding whether a technical justification is valid is
>>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>>> > this vote. This is neither a vote on a code change itself not a vote
>>>>> > on whether a package is ready for release, so it a procedural vote on
>>>>> > whether the technical justification is valid. As such, the vote will
>>>>> > be decided by a simple majority where +1 votes hold that the
>>>>> technical
>>>>> > justification is not valid and -1 votes hold that the technical
>>>>> > justification is valid.
>>>>> >
>>>>> > I would request that at least PMC members post more than just a naked
>>>>> > vote, but instead endeavor to give some reason why they have assessed
>>>>> > the technical justification as they have. I'll start:
>>>>> >
>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>>>> > confess to still not being entirely clear on what is his technical
>>>>> > justification for that veto. I see claims that including an
>>>>> admonition
>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>>> > required to maintain the integrity of already existing data streams,
>>>>> > and I see assertions about the maintenance burden that including the
>>>>> > migration logic would impose on future Spark versions, but I don't
>>>>> > think that I see any other technical objections. I do not believe
>>>>> that
>>>>> > the claimed technical justification is valid.
>>>>> >
>>>>> > In requiring that a veto of a code change be accompanied by a
>>>>> > technical justification for the veto, the Apache Voting Process
>>>>> states
>>>>> > that: "To prevent vetoes from being used capriciously, the voter must
>>>>> > provide with the veto a technical justification showing why the
>>>>> change
>>>>> > is bad (opens a security exposure, negatively affects performance,
>>>>> > etc. ). A veto without a justification is invalid and has no weight."
>>>>> > This strongly implies that there must be something objectively wrong
>>>>> > with the proposed code change in that it causes significant harm in
>>>>> > the way of opening a security exposure, negatively affecting
>>>>> > performance, or presumably other significant user harms or perhaps
>>>>> > even developer burdens.
>>>>> >
>>>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>>>> > cause any harm to Spark's users. For many users, those not using
>>>>> > streaming data, the change will have no effect. For streaming users
>>>>> > the change will be beneficial, not harmful.
>>>>> >
>>>>> > Neither do I find the claim of excessive, ongoing developer burden to
>>>>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>>>>> > it wouldn't surprise me if no further changes to this migration logic
>>>>> > would be needed for a very long time.
>>>>> >
>>>>> > Some of what we are left with is just an expression of preference for
>>>>> > a technical alternative to the migration logic -- i.e. including in
>>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>>> > Apache Voting Process does not say that in the face of code
>>>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>>> > something more is needed to justify a veto, as I've already covered.
>>>>> > Thus I don't find Dongjoon's preference for the release notes option
>>>>> > to be adequate justification for the veto.
>>>>> >
>>>>> > The only remaining question I see is whether including "databricks"
>>>>> in
>>>>> > the Apache Code is ever allowed or if any such instance must be
>>>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>>> > reason, even if that vendor has a product based on Apache code, even
>>>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>>>> > Apache code or project. Certainly the PMC has a duty to see to it
>>>>> that
>>>>> > neither Databricks nor any other vendor exercises influence or
>>>>> control
>>>>> > over Apache Spark outside of the established Apache process, but the
>>>>> > proposed migration code changes do not advantage Databricks -- if
>>>>> > anything they remove a minor avenue of influence, and simply need to
>>>>> > mention "databricks" once in order match and transform a
>>>>> configuration
>>>>> > into a vendor neutral equivalent. While not optimal, I can't find
>>>>> such
>>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>>> > non-technical policy concern -- certainly not offensive to the point
>>>>> > that it outweighs the user advantage of including the migration logic
>>>>> > in Spark 4.0.x.
>>>>> >
>>>>> > In summary, I do not find Dongjoon's given technical justification to
>>>>> > be valid relative to the Apache requirements for a veto of a code
>>>>> > change, so I must vote...
>>>>> >
>>>>> > +1
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Holden Karau <ho...@gmail.com>]

I believe he intended to VETO this change for Spark 4, of course if
Dongjoon did not (or no longer intends to) then this VOTE becomes moot. I
think bringing up 3.5.5 confuses the issue -- this vote thread is very
clearly about VETOing the change for Spark 4.

I think that accusing each-other of acting in bad faith is unproductive to
resolving this dispute.

On Sun, Mar 16, 2025 at 3:14 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Holden and Dongjoon,
>
> Let me make this vote super simple. I never got the answer from Dongjoon
> about this question. This is super important because if he's casting veto
> "to block", it is a strong indication that he was intended to play with me,
> which I am seriously considering escalating the problem (If this is true,
> it's no longer just a justification of vote, but someone's
> behavioral issue).
>
> https://github.com/apache/spark/pull/49983
>
> I might be missing another timeline, but, if you follow the conversation
> here, there are some facts:
>
> 1. Dongjoon "knew" we were never decided about the direction of Spark
> 4.0.0 behavior. (link
> <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>)
> 2. Dongjoon "agreed" my proposal is technically correct. (link
> <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>)
> 3. Dongjoon "agreed" to hear from the community about discussing my
> proposal. (link
> <https://github.com/apache/spark/pull/49983#issuecomment-2696064545>)
>
> Worth clarifying, 3 happened after we discussed the removal of "config".
> Dongjoon continuously mixed up the fact - while we were in agreement of
> removal of config, removal of migration logic was definitely left to open
> question. Let me give the VOTE Dongjoon drove and made it pass.
>
> https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5
>
> This was totally about 3.5.5. If Dongjoon thinks this simply applies to
> Spark 4.0.0+, it's not, no?
>
> Also, let's revisit the discussion we were discussing about removal of
> config.
>
> https://lists.apache.org/thread/qxqzt7wbtdyxp17d7s1rxhnrswdccsgb
>
> Dongjoon clearly stated that we only make a consensus about Spark 3.5.5,
> and we can continue discussion about the proper behavior in Spark 4.0.0.
> That is the rationale I drove my own discussion. I can be corrected, but
> there is NO discussion/vote w.r.t this topic AFAIK.
>
> Dongjoon, now it's your time to prove there is a valid reason to change
> your mind during this time frame. If the above are all true, you are
> already indicating that you can never cast a veto. (Or show me the evidence
> of how you change your mind for which reason.) If any of the above are
> something you intended to not tell the truth, I am really not sure your
> comment will be truthful I can follow. Especially, if you did not tell the
> truth from 3, e.g. you let me go and discuss while you were intended to
> block me in any phase, this is a strong indication that you intend to play
> with me and the community (or even ASF) has to know that.
>
> Do not evade the root question.
>
> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com>
> wrote:
>
>> -1 (binding) — to me it doesn’t matter that the cost is low if the
>> objection is technical then I think we need to respect the veto. There is a
>> fundamental disagreement as to what the correct technical way to address
>> this problem is (removal + documentation vs legacy config) and a PMC member
>> has vetoed  the legacy config option.
>>
>> I think I disagree with Mark on the assertion that the veto needs to have
>> “substantial technical concern,” but rather a valid concern. I think in
>> addition to the veto they’ve also gone above and beyond providing
>> alternative ways to accomplish this.
>>
>> On a personal level:
>>
>> I am optimistic we can unblock the release but I think it’s important to
>> err on the side of respecting the veto here in the interest of perceived
>> fairness *especially* because of vendor aspects.
>>
>> To be clear I’ve worked at most of these companies (and many of the
>> people) and I’m not ascribing malice to anyone in this, I think mistakes
>> happen (god knows I’ve had a fair share). I think we’re all doing our best
>> here and would ask that we show everyone understanding regardless of the
>> outcome.
>>
>> Sending hugs and good vibes to y’all.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> Given it’s the weekend maybe let’s give folks at least one full work day.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>>> wrote:
>>>
>>>> Quick administrative note: I don't see any reason why this vote should
>>>> take a long time, so I expect to close the process and tally the votes
>>>> in not much more than 48 hours.
>>>>
>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>>> wrote:
>>>> >
>>>> > There has been enough discussion on this topic already, so I think
>>>> > that an immediate vote on the validity of Dongjoon's technical
>>>> > justification for his veto of the "Retain migration logic ... in Spark
>>>> > 4.0.x" proposal is in order. That technical justification has been
>>>> > called into question, and the guidance at
>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>>>> > PMC to determine whether the technical justification is  valid: "In
>>>> > case of doubt, deciding whether a technical justification is valid is
>>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>>> > this vote. This is neither a vote on a code change itself not a vote
>>>> > on whether a package is ready for release, so it a procedural vote on
>>>> > whether the technical justification is valid. As such, the vote will
>>>> > be decided by a simple majority where +1 votes hold that the technical
>>>> > justification is not valid and -1 votes hold that the technical
>>>> > justification is valid.
>>>> >
>>>> > I would request that at least PMC members post more than just a naked
>>>> > vote, but instead endeavor to give some reason why they have assessed
>>>> > the technical justification as they have. I'll start:
>>>> >
>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>>> > confess to still not being entirely clear on what is his technical
>>>> > justification for that veto. I see claims that including an admonition
>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>>> > required to maintain the integrity of already existing data streams,
>>>> > and I see assertions about the maintenance burden that including the
>>>> > migration logic would impose on future Spark versions, but I don't
>>>> > think that I see any other technical objections. I do not believe that
>>>> > the claimed technical justification is valid.
>>>> >
>>>> > In requiring that a veto of a code change be accompanied by a
>>>> > technical justification for the veto, the Apache Voting Process states
>>>> > that: "To prevent vetoes from being used capriciously, the voter must
>>>> > provide with the veto a technical justification showing why the change
>>>> > is bad (opens a security exposure, negatively affects performance,
>>>> > etc. ). A veto without a justification is invalid and has no weight."
>>>> > This strongly implies that there must be something objectively wrong
>>>> > with the proposed code change in that it causes significant harm in
>>>> > the way of opening a security exposure, negatively affecting
>>>> > performance, or presumably other significant user harms or perhaps
>>>> > even developer burdens.
>>>> >
>>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>>> > cause any harm to Spark's users. For many users, those not using
>>>> > streaming data, the change will have no effect. For streaming users
>>>> > the change will be beneficial, not harmful.
>>>> >
>>>> > Neither do I find the claim of excessive, ongoing developer burden to
>>>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>>>> > it wouldn't surprise me if no further changes to this migration logic
>>>> > would be needed for a very long time.
>>>> >
>>>> > Some of what we are left with is just an expression of preference for
>>>> > a technical alternative to the migration logic -- i.e. including in
>>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>>> > Apache Voting Process does not say that in the face of code
>>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>>> > they prefer B. Instead, the Voting Process strongly implies that
>>>> > something more is needed to justify a veto, as I've already covered.
>>>> > Thus I don't find Dongjoon's preference for the release notes option
>>>> > to be adequate justification for the veto.
>>>> >
>>>> > The only remaining question I see is whether including "databricks" in
>>>> > the Apache Code is ever allowed or if any such instance must be
>>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>>> > strictly forbids the mention of a vendor in Apache code for any
>>>> > reason, even if that vendor has a product based on Apache code, even
>>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>>> > Apache code or project. Certainly the PMC has a duty to see to it that
>>>> > neither Databricks nor any other vendor exercises influence or control
>>>> > over Apache Spark outside of the established Apache process, but the
>>>> > proposed migration code changes do not advantage Databricks -- if
>>>> > anything they remove a minor avenue of influence, and simply need to
>>>> > mention "databricks" once in order match and transform a configuration
>>>> > into a vendor neutral equivalent. While not optimal, I can't find such
>>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>>> > non-technical policy concern -- certainly not offensive to the point
>>>> > that it outweighs the user advantage of including the migration logic
>>>> > in Spark 4.0.x.
>>>> >
>>>> > In summary, I do not find Dongjoon's given technical justification to
>>>> > be valid relative to the Apache requirements for a veto of a code
>>>> > change, so I must vote...
>>>> >
>>>> > +1
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>

-- 
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Holden and Dongjoon,

Let me make this vote super simple. I never got the answer from Dongjoon
about this question. This is super important because if he's casting veto
"to block", it is a strong indication that he was intended to play with me,
which I am seriously considering escalating the problem (If this is true,
it's no longer just a justification of vote, but someone's
behavioral issue).

https://github.com/apache/spark/pull/49983

I might be missing another timeline, but, if you follow the conversation
here, there are some facts:

1. Dongjoon "knew" we were never decided about the direction of Spark 4.0.0
behavior. (link
<https://github.com/apache/spark/pull/49983#issuecomment-2676531485>)
2. Dongjoon "agreed" my proposal is technically correct. (link
<https://github.com/apache/spark/pull/49983#issuecomment-2676531485>)
3. Dongjoon "agreed" to hear from the community about discussing my
proposal. (link
<https://github.com/apache/spark/pull/49983#issuecomment-2696064545>)

Worth clarifying, 3 happened after we discussed the removal of "config".
Dongjoon continuously mixed up the fact - while we were in agreement of
removal of config, removal of migration logic was definitely left to open
question. Let me give the VOTE Dongjoon drove and made it pass.

https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5

This was totally about 3.5.5. If Dongjoon thinks this simply applies to
Spark 4.0.0+, it's not, no?

Also, let's revisit the discussion we were discussing about removal of
config.

https://lists.apache.org/thread/qxqzt7wbtdyxp17d7s1rxhnrswdccsgb

Dongjoon clearly stated that we only make a consensus about Spark 3.5.5,
and we can continue discussion about the proper behavior in Spark 4.0.0.
That is the rationale I drove my own discussion. I can be corrected, but
there is NO discussion/vote w.r.t this topic AFAIK.

Dongjoon, now it's your time to prove there is a valid reason to change
your mind during this time frame. If the above are all true, you are
already indicating that you can never cast a veto. (Or show me the evidence
of how you change your mind for which reason.) If any of the above are
something you intended to not tell the truth, I am really not sure your
comment will be truthful I can follow. Especially, if you did not tell the
truth from 3, e.g. you let me go and discuss while you were intended to
block me in any phase, this is a strong indication that you intend to play
with me and the community (or even ASF) has to know that.

Do not evade the root question.

On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <ho...@gmail.com> wrote:

> -1 (binding) — to me it doesn’t matter that the cost is low if the
> objection is technical then I think we need to respect the veto. There is a
> fundamental disagreement as to what the correct technical way to address
> this problem is (removal + documentation vs legacy config) and a PMC member
> has vetoed  the legacy config option.
>
> I think I disagree with Mark on the assertion that the veto needs to have
> “substantial technical concern,” but rather a valid concern. I think in
> addition to the veto they’ve also gone above and beyond providing
> alternative ways to accomplish this.
>
> On a personal level:
>
> I am optimistic we can unblock the release but I think it’s important to
> err on the side of respecting the veto here in the interest of perceived
> fairness *especially* because of vendor aspects.
>
> To be clear I’ve worked at most of these companies (and many of the
> people) and I’m not ascribing malice to anyone in this, I think mistakes
> happen (god knows I’ve had a fair share). I think we’re all doing our best
> here and would ask that we show everyone understanding regardless of the
> outcome.
>
> Sending hugs and good vibes to y’all.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
> wrote:
>
>> Given it’s the weekend maybe let’s give folks at least one full work day.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Quick administrative note: I don't see any reason why this vote should
>>> take a long time, so I expect to close the process and tally the votes
>>> in not much more than 48 hours.
>>>
>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>> wrote:
>>> >
>>> > There has been enough discussion on this topic already, so I think
>>> > that an immediate vote on the validity of Dongjoon's technical
>>> > justification for his veto of the "Retain migration logic ... in Spark
>>> > 4.0.x" proposal is in order. That technical justification has been
>>> > called into question, and the guidance at
>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>>> > PMC to determine whether the technical justification is  valid: "In
>>> > case of doubt, deciding whether a technical justification is valid is
>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>> > this vote. This is neither a vote on a code change itself not a vote
>>> > on whether a package is ready for release, so it a procedural vote on
>>> > whether the technical justification is valid. As such, the vote will
>>> > be decided by a simple majority where +1 votes hold that the technical
>>> > justification is not valid and -1 votes hold that the technical
>>> > justification is valid.
>>> >
>>> > I would request that at least PMC members post more than just a naked
>>> > vote, but instead endeavor to give some reason why they have assessed
>>> > the technical justification as they have. I'll start:
>>> >
>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>> > confess to still not being entirely clear on what is his technical
>>> > justification for that veto. I see claims that including an admonition
>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>> > required to maintain the integrity of already existing data streams,
>>> > and I see assertions about the maintenance burden that including the
>>> > migration logic would impose on future Spark versions, but I don't
>>> > think that I see any other technical objections. I do not believe that
>>> > the claimed technical justification is valid.
>>> >
>>> > In requiring that a veto of a code change be accompanied by a
>>> > technical justification for the veto, the Apache Voting Process states
>>> > that: "To prevent vetoes from being used capriciously, the voter must
>>> > provide with the veto a technical justification showing why the change
>>> > is bad (opens a security exposure, negatively affects performance,
>>> > etc. ). A veto without a justification is invalid and has no weight."
>>> > This strongly implies that there must be something objectively wrong
>>> > with the proposed code change in that it causes significant harm in
>>> > the way of opening a security exposure, negatively affecting
>>> > performance, or presumably other significant user harms or perhaps
>>> > even developer burdens.
>>> >
>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>> > cause any harm to Spark's users. For many users, those not using
>>> > streaming data, the change will have no effect. For streaming users
>>> > the change will be beneficial, not harmful.
>>> >
>>> > Neither do I find the claim of excessive, ongoing developer burden to
>>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>>> > it wouldn't surprise me if no further changes to this migration logic
>>> > would be needed for a very long time.
>>> >
>>> > Some of what we are left with is just an expression of preference for
>>> > a technical alternative to the migration logic -- i.e. including in
>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>> > Apache Voting Process does not say that in the face of code
>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>> > they prefer B. Instead, the Voting Process strongly implies that
>>> > something more is needed to justify a veto, as I've already covered.
>>> > Thus I don't find Dongjoon's preference for the release notes option
>>> > to be adequate justification for the veto.
>>> >
>>> > The only remaining question I see is whether including "databricks" in
>>> > the Apache Code is ever allowed or if any such instance must be
>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>> > strictly forbids the mention of a vendor in Apache code for any
>>> > reason, even if that vendor has a product based on Apache code, even
>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>> > Apache code or project. Certainly the PMC has a duty to see to it that
>>> > neither Databricks nor any other vendor exercises influence or control
>>> > over Apache Spark outside of the established Apache process, but the
>>> > proposed migration code changes do not advantage Databricks -- if
>>> > anything they remove a minor avenue of influence, and simply need to
>>> > mention "databricks" once in order match and transform a configuration
>>> > into a vendor neutral equivalent. While not optimal, I can't find such
>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>> > non-technical policy concern -- certainly not offensive to the point
>>> > that it outweighs the user advantage of including the migration logic
>>> > in Spark 4.0.x.
>>> >
>>> > In summary, I do not find Dongjoon's given technical justification to
>>> > be valid relative to the Apache requirements for a veto of a code
>>> > change, so I must vote...
>>> >
>>> > +1
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Mich Talebzadeh <mi...@gmail.com>]

I second this point of Holden.

In fairness anything published in a public forum like this one is fair game
for analysis or criticism. That is the practice of analyzing, classifying,
interpreting, or evaluating the technical matter. If someone makes a claim
on a technical or procedure matter in an open forum, then the person is
expected to back it up. *I cannot see how anyone could object to the
statement: if you make a claim or have a strong opinion, be prepared to
prove it or debate it.* Regardless, as stated mistakes can and do happen.

 HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR

   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>





On Sun, 16 Mar 2025 at 21:32, Holden Karau <ho...@gmail.com> wrote:

> -1 (binding) — to me it doesn’t matter that the cost is low if the
> objection is technical then I think we need to respect the veto. There is a
> fundamental disagreement as to what the correct technical way to address
> this problem is (removal + documentation vs legacy config) and a PMC member
> has vetoed  the legacy config option.
>
> I think I disagree with Mark on the assertion that the veto needs to have
> “substantial technical concern,” but rather a valid concern. I think in
> addition to the veto they’ve also gone above and beyond providing
> alternative ways to accomplish this.
>
> On a personal level:
>
> I am optimistic we can unblock the release but I think it’s important to
> err on the side of respecting the veto here in the interest of perceived
> fairness *especially* because of vendor aspects.
>
> To be clear I’ve worked at most of these companies (and many of the
> people) and I’m not ascribing malice to anyone in this, I think mistakes
> happen (god knows I’ve had a fair share). I think we’re all doing our best
> here and would ask that we show everyone understanding regardless of the
> outcome.
>
> Sending hugs and good vibes to y’all.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com>
> wrote:
>
>> Given it’s the weekend maybe let’s give folks at least one full work day.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>>
>>> Quick administrative note: I don't see any reason why this vote should
>>> take a long time, so I expect to close the process and tally the votes
>>> in not much more than 48 hours.
>>>
>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>>> wrote:
>>> >
>>> > There has been enough discussion on this topic already, so I think
>>> > that an immediate vote on the validity of Dongjoon's technical
>>> > justification for his veto of the "Retain migration logic ... in Spark
>>> > 4.0.x" proposal is in order. That technical justification has been
>>> > called into question, and the guidance at
>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>>> > PMC to determine whether the technical justification is  valid: "In
>>> > case of doubt, deciding whether a technical justification is valid is
>>> > up to the PMC." As such, only PMC votes will decide the outcome of
>>> > this vote. This is neither a vote on a code change itself not a vote
>>> > on whether a package is ready for release, so it a procedural vote on
>>> > whether the technical justification is valid. As such, the vote will
>>> > be decided by a simple majority where +1 votes hold that the technical
>>> > justification is not valid and -1 votes hold that the technical
>>> > justification is valid.
>>> >
>>> > I would request that at least PMC members post more than just a naked
>>> > vote, but instead endeavor to give some reason why they have assessed
>>> > the technical justification as they have. I'll start:
>>> >
>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>>> > confess to still not being entirely clear on what is his technical
>>> > justification for that veto. I see claims that including an admonition
>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>>> > required to maintain the integrity of already existing data streams,
>>> > and I see assertions about the maintenance burden that including the
>>> > migration logic would impose on future Spark versions, but I don't
>>> > think that I see any other technical objections. I do not believe that
>>> > the claimed technical justification is valid.
>>> >
>>> > In requiring that a veto of a code change be accompanied by a
>>> > technical justification for the veto, the Apache Voting Process states
>>> > that: "To prevent vetoes from being used capriciously, the voter must
>>> > provide with the veto a technical justification showing why the change
>>> > is bad (opens a security exposure, negatively affects performance,
>>> > etc. ). A veto without a justification is invalid and has no weight."
>>> > This strongly implies that there must be something objectively wrong
>>> > with the proposed code change in that it causes significant harm in
>>> > the way of opening a security exposure, negatively affecting
>>> > performance, or presumably other significant user harms or perhaps
>>> > even developer burdens.
>>> >
>>> > The proposed addition of the migration logic to Spark 4.0.x does not
>>> > cause any harm to Spark's users. For many users, those not using
>>> > streaming data, the change will have no effect. For streaming users
>>> > the change will be beneficial, not harmful.
>>> >
>>> > Neither do I find the claim of excessive, ongoing developer burden to
>>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>>> > it wouldn't surprise me if no further changes to this migration logic
>>> > would be needed for a very long time.
>>> >
>>> > Some of what we are left with is just an expression of preference for
>>> > a technical alternative to the migration logic -- i.e. including in
>>> > the release notes an admonition to first upgrade to 3.5.5. But the
>>> > Apache Voting Process does not say that in the face of code
>>> > alternatives A and B, a qualified voter is justified in vetoing A if
>>> > they prefer B. Instead, the Voting Process strongly implies that
>>> > something more is needed to justify a veto, as I've already covered.
>>> > Thus I don't find Dongjoon's preference for the release notes option
>>> > to be adequate justification for the veto.
>>> >
>>> > The only remaining question I see is whether including "databricks" in
>>> > the Apache Code is ever allowed or if any such instance must be
>>> > expunged as soon as possible. I am not aware of any ASF policy that
>>> > strictly forbids the mention of a vendor in Apache code for any
>>> > reason, even if that vendor has a product based on Apache code, even
>>> > if that vendor enjoys a uniquely influential position vis a vis some
>>> > Apache code or project. Certainly the PMC has a duty to see to it that
>>> > neither Databricks nor any other vendor exercises influence or control
>>> > over Apache Spark outside of the established Apache process, but the
>>> > proposed migration code changes do not advantage Databricks -- if
>>> > anything they remove a minor avenue of influence, and simply need to
>>> > mention "databricks" once in order match and transform a configuration
>>> > into a vendor neutral equivalent. While not optimal, I can't find such
>>> > a one-time inclusion of "databricks" to be truly offensive to any
>>> > non-technical policy concern -- certainly not offensive to the point
>>> > that it outweighs the user advantage of including the migration logic
>>> > in Spark 4.0.x.
>>> >
>>> > In summary, I do not find Dongjoon's given technical justification to
>>> > be valid relative to the Apache requirements for a veto of a code
>>> > change, so I must vote...
>>> >
>>> > +1
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Holden Karau <ho...@gmail.com>]

-1 (binding) — to me it doesn’t matter that the cost is low if the
objection is technical then I think we need to respect the veto. There is a
fundamental disagreement as to what the correct technical way to address
this problem is (removal + documentation vs legacy config) and a PMC member
has vetoed  the legacy config option.

I think I disagree with Mark on the assertion that the veto needs to have
“substantial technical concern,” but rather a valid concern. I think in
addition to the veto they’ve also gone above and beyond providing
alternative ways to accomplish this.

On a personal level:

I am optimistic we can unblock the release but I think it’s important to
err on the side of respecting the veto here in the interest of perceived
fairness *especially* because of vendor aspects.

To be clear I’ve worked at most of these companies (and many of the people)
and I’m not ascribing malice to anyone in this, I think mistakes happen
(god knows I’ve had a fair share). I think we’re all doing our best here
and would ask that we show everyone understanding regardless of the outcome.

Sending hugs and good vibes to y’all.

Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her


On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <ho...@gmail.com> wrote:

> Given it’s the weekend maybe let’s give folks at least one full work day.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com>
> wrote:
>
>> Quick administrative note: I don't see any reason why this vote should
>> take a long time, so I expect to close the process and tally the votes
>> in not much more than 48 hours.
>>
>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
>> wrote:
>> >
>> > There has been enough discussion on this topic already, so I think
>> > that an immediate vote on the validity of Dongjoon's technical
>> > justification for his veto of the "Retain migration logic ... in Spark
>> > 4.0.x" proposal is in order. That technical justification has been
>> > called into question, and the guidance at
>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
>> > PMC to determine whether the technical justification is  valid: "In
>> > case of doubt, deciding whether a technical justification is valid is
>> > up to the PMC." As such, only PMC votes will decide the outcome of
>> > this vote. This is neither a vote on a code change itself not a vote
>> > on whether a package is ready for release, so it a procedural vote on
>> > whether the technical justification is valid. As such, the vote will
>> > be decided by a simple majority where +1 votes hold that the technical
>> > justification is not valid and -1 votes hold that the technical
>> > justification is valid.
>> >
>> > I would request that at least PMC members post more than just a naked
>> > vote, but instead endeavor to give some reason why they have assessed
>> > the technical justification as they have. I'll start:
>> >
>> > Despite all of the discussion related to Dongjoon's -1 vote, I must
>> > confess to still not being entirely clear on what is his technical
>> > justification for that veto. I see claims that including an admonition
>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
>> > required to maintain the integrity of already existing data streams,
>> > and I see assertions about the maintenance burden that including the
>> > migration logic would impose on future Spark versions, but I don't
>> > think that I see any other technical objections. I do not believe that
>> > the claimed technical justification is valid.
>> >
>> > In requiring that a veto of a code change be accompanied by a
>> > technical justification for the veto, the Apache Voting Process states
>> > that: "To prevent vetoes from being used capriciously, the voter must
>> > provide with the veto a technical justification showing why the change
>> > is bad (opens a security exposure, negatively affects performance,
>> > etc. ). A veto without a justification is invalid and has no weight."
>> > This strongly implies that there must be something objectively wrong
>> > with the proposed code change in that it causes significant harm in
>> > the way of opening a security exposure, negatively affecting
>> > performance, or presumably other significant user harms or perhaps
>> > even developer burdens.
>> >
>> > The proposed addition of the migration logic to Spark 4.0.x does not
>> > cause any harm to Spark's users. For many users, those not using
>> > streaming data, the change will have no effect. For streaming users
>> > the change will be beneficial, not harmful.
>> >
>> > Neither do I find the claim of excessive, ongoing developer burden to
>> > be persuasive. The changes are tiny and easily maintained -- in fact,
>> > it wouldn't surprise me if no further changes to this migration logic
>> > would be needed for a very long time.
>> >
>> > Some of what we are left with is just an expression of preference for
>> > a technical alternative to the migration logic -- i.e. including in
>> > the release notes an admonition to first upgrade to 3.5.5. But the
>> > Apache Voting Process does not say that in the face of code
>> > alternatives A and B, a qualified voter is justified in vetoing A if
>> > they prefer B. Instead, the Voting Process strongly implies that
>> > something more is needed to justify a veto, as I've already covered.
>> > Thus I don't find Dongjoon's preference for the release notes option
>> > to be adequate justification for the veto.
>> >
>> > The only remaining question I see is whether including "databricks" in
>> > the Apache Code is ever allowed or if any such instance must be
>> > expunged as soon as possible. I am not aware of any ASF policy that
>> > strictly forbids the mention of a vendor in Apache code for any
>> > reason, even if that vendor has a product based on Apache code, even
>> > if that vendor enjoys a uniquely influential position vis a vis some
>> > Apache code or project. Certainly the PMC has a duty to see to it that
>> > neither Databricks nor any other vendor exercises influence or control
>> > over Apache Spark outside of the established Apache process, but the
>> > proposed migration code changes do not advantage Databricks -- if
>> > anything they remove a minor avenue of influence, and simply need to
>> > mention "databricks" once in order match and transform a configuration
>> > into a vendor neutral equivalent. While not optimal, I can't find such
>> > a one-time inclusion of "databricks" to be truly offensive to any
>> > non-technical policy concern -- certainly not offensive to the point
>> > that it outweighs the user advantage of including the migration logic
>> > in Spark 4.0.x.
>> >
>> > In summary, I do not find Dongjoon's given technical justification to
>> > be valid relative to the Apache requirements for a veto of a code
>> > change, so I must vote...
>> >
>> > +1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Holden Karau <ho...@gmail.com>]

Given it’s the weekend maybe let’s give folks at least one full work day.

Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her


On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <ma...@gmail.com> wrote:

> Quick administrative note: I don't see any reason why this vote should
> take a long time, so I expect to close the process and tally the votes
> in not much more than 48 hours.
>
> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com>
> wrote:
> >
> > There has been enough discussion on this topic already, so I think
> > that an immediate vote on the validity of Dongjoon's technical
> > justification for his veto of the "Retain migration logic ... in Spark
> > 4.0.x" proposal is in order. That technical justification has been
> > called into question, and the guidance at
> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the
> > PMC to determine whether the technical justification is  valid: "In
> > case of doubt, deciding whether a technical justification is valid is
> > up to the PMC." As such, only PMC votes will decide the outcome of
> > this vote. This is neither a vote on a code change itself not a vote
> > on whether a package is ready for release, so it a procedural vote on
> > whether the technical justification is valid. As such, the vote will
> > be decided by a simple majority where +1 votes hold that the technical
> > justification is not valid and -1 votes hold that the technical
> > justification is valid.
> >
> > I would request that at least PMC members post more than just a naked
> > vote, but instead endeavor to give some reason why they have assessed
> > the technical justification as they have. I'll start:
> >
> > Despite all of the discussion related to Dongjoon's -1 vote, I must
> > confess to still not being entirely clear on what is his technical
> > justification for that veto. I see claims that including an admonition
> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
> > required to maintain the integrity of already existing data streams,
> > and I see assertions about the maintenance burden that including the
> > migration logic would impose on future Spark versions, but I don't
> > think that I see any other technical objections. I do not believe that
> > the claimed technical justification is valid.
> >
> > In requiring that a veto of a code change be accompanied by a
> > technical justification for the veto, the Apache Voting Process states
> > that: "To prevent vetoes from being used capriciously, the voter must
> > provide with the veto a technical justification showing why the change
> > is bad (opens a security exposure, negatively affects performance,
> > etc. ). A veto without a justification is invalid and has no weight."
> > This strongly implies that there must be something objectively wrong
> > with the proposed code change in that it causes significant harm in
> > the way of opening a security exposure, negatively affecting
> > performance, or presumably other significant user harms or perhaps
> > even developer burdens.
> >
> > The proposed addition of the migration logic to Spark 4.0.x does not
> > cause any harm to Spark's users. For many users, those not using
> > streaming data, the change will have no effect. For streaming users
> > the change will be beneficial, not harmful.
> >
> > Neither do I find the claim of excessive, ongoing developer burden to
> > be persuasive. The changes are tiny and easily maintained -- in fact,
> > it wouldn't surprise me if no further changes to this migration logic
> > would be needed for a very long time.
> >
> > Some of what we are left with is just an expression of preference for
> > a technical alternative to the migration logic -- i.e. including in
> > the release notes an admonition to first upgrade to 3.5.5. But the
> > Apache Voting Process does not say that in the face of code
> > alternatives A and B, a qualified voter is justified in vetoing A if
> > they prefer B. Instead, the Voting Process strongly implies that
> > something more is needed to justify a veto, as I've already covered.
> > Thus I don't find Dongjoon's preference for the release notes option
> > to be adequate justification for the veto.
> >
> > The only remaining question I see is whether including "databricks" in
> > the Apache Code is ever allowed or if any such instance must be
> > expunged as soon as possible. I am not aware of any ASF policy that
> > strictly forbids the mention of a vendor in Apache code for any
> > reason, even if that vendor has a product based on Apache code, even
> > if that vendor enjoys a uniquely influential position vis a vis some
> > Apache code or project. Certainly the PMC has a duty to see to it that
> > neither Databricks nor any other vendor exercises influence or control
> > over Apache Spark outside of the established Apache process, but the
> > proposed migration code changes do not advantage Databricks -- if
> > anything they remove a minor avenue of influence, and simply need to
> > mention "databricks" once in order match and transform a configuration
> > into a vendor neutral equivalent. While not optimal, I can't find such
> > a one-time inclusion of "databricks" to be truly offensive to any
> > non-technical policy concern -- certainly not offensive to the point
> > that it outweighs the user advantage of including the migration logic
> > in Spark 4.0.x.
> >
> > In summary, I do not find Dongjoon's given technical justification to
> > be valid relative to the Apache requirements for a veto of a code
> > change, so I must vote...
> >
> > +1
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Mark Hamstra <ma...@gmail.com>]

Quick administrative note: I don't see any reason why this vote should
take a long time, so I expect to close the process and tally the votes
in not much more than 48 hours.

On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <ma...@gmail.com> wrote:
>
> There has been enough discussion on this topic already, so I think
> that an immediate vote on the validity of Dongjoon's technical
> justification for his veto of the "Retain migration logic ... in Spark
> 4.0.x" proposal is in order. That technical justification has been
> called into question, and the guidance at
> https://www.apache.org/foundation/glossary.html#Veto leaves it to the
> PMC to determine whether the technical justification is  valid: "In
> case of doubt, deciding whether a technical justification is valid is
> up to the PMC." As such, only PMC votes will decide the outcome of
> this vote. This is neither a vote on a code change itself not a vote
> on whether a package is ready for release, so it a procedural vote on
> whether the technical justification is valid. As such, the vote will
> be decided by a simple majority where +1 votes hold that the technical
> justification is not valid and -1 votes hold that the technical
> justification is valid.
>
> I would request that at least PMC members post more than just a naked
> vote, but instead endeavor to give some reason why they have assessed
> the technical justification as they have. I'll start:
>
> Despite all of the discussion related to Dongjoon's -1 vote, I must
> confess to still not being entirely clear on what is his technical
> justification for that veto. I see claims that including an admonition
> in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is
> required to maintain the integrity of already existing data streams,
> and I see assertions about the maintenance burden that including the
> migration logic would impose on future Spark versions, but I don't
> think that I see any other technical objections. I do not believe that
> the claimed technical justification is valid.
>
> In requiring that a veto of a code change be accompanied by a
> technical justification for the veto, the Apache Voting Process states
> that: "To prevent vetoes from being used capriciously, the voter must
> provide with the veto a technical justification showing why the change
> is bad (opens a security exposure, negatively affects performance,
> etc. ). A veto without a justification is invalid and has no weight."
> This strongly implies that there must be something objectively wrong
> with the proposed code change in that it causes significant harm in
> the way of opening a security exposure, negatively affecting
> performance, or presumably other significant user harms or perhaps
> even developer burdens.
>
> The proposed addition of the migration logic to Spark 4.0.x does not
> cause any harm to Spark's users. For many users, those not using
> streaming data, the change will have no effect. For streaming users
> the change will be beneficial, not harmful.
>
> Neither do I find the claim of excessive, ongoing developer burden to
> be persuasive. The changes are tiny and easily maintained -- in fact,
> it wouldn't surprise me if no further changes to this migration logic
> would be needed for a very long time.
>
> Some of what we are left with is just an expression of preference for
> a technical alternative to the migration logic -- i.e. including in
> the release notes an admonition to first upgrade to 3.5.5. But the
> Apache Voting Process does not say that in the face of code
> alternatives A and B, a qualified voter is justified in vetoing A if
> they prefer B. Instead, the Voting Process strongly implies that
> something more is needed to justify a veto, as I've already covered.
> Thus I don't find Dongjoon's preference for the release notes option
> to be adequate justification for the veto.
>
> The only remaining question I see is whether including "databricks" in
> the Apache Code is ever allowed or if any such instance must be
> expunged as soon as possible. I am not aware of any ASF policy that
> strictly forbids the mention of a vendor in Apache code for any
> reason, even if that vendor has a product based on Apache code, even
> if that vendor enjoys a uniquely influential position vis a vis some
> Apache code or project. Certainly the PMC has a duty to see to it that
> neither Databricks nor any other vendor exercises influence or control
> over Apache Spark outside of the established Apache process, but the
> proposed migration code changes do not advantage Databricks -- if
> anything they remove a minor avenue of influence, and simply need to
> mention "databricks" once in order match and transform a configuration
> into a vendor neutral equivalent. While not optimal, I can't find such
> a one-time inclusion of "databricks" to be truly offensive to any
> non-technical policy concern -- certainly not offensive to the point
> that it outweighs the user advantage of including the migration logic
> in Spark 4.0.x.
>
> In summary, I do not find Dongjoon's given technical justification to
> be valid relative to the Apache requirements for a veto of a code
> change, so I must vote...
>
> +1

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Just one hope, I believe I have said I will hear about support of migration
logic for next release. The scope of the VOTE is nothing beyond 4.0.x.
Please do not interpret this in your own way. I continuously see that I am
attacked by what I am not saying and I have to prove that I haven't said.
Please stop it.

On Sun, Mar 16, 2025 at 3:14 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Dongjoon,
>
> I'm now OK with whatever you think, but I argue your vote is technically
> moot since it's about your vote justification, and I have no binding vote
> to counter you. Let's be fair.
>
> On Sun, Mar 16, 2025 at 3:07 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> Thank you for focusing on this, Mark.
>>
>> I also agree with you that this should be decided by the Apache Spark PMC
>> and appreciate the effort to help us move forward in the Apache way.
>>
>> As you mentioned, there is no ASF policy. That's true.
>>
>> > I am not aware of any ASF policy that strictly forbids the mention of a
>> vendor
>> > in Apache code for any reason
>>
>> Let's imagine that Apache Spark project starts to support the following
>> existing vendor `spark.databricks.*` configs to help Spark users migrate
>> (or offload) from Databricks service to open source Spark cluster easily.
>>
>> - spark.databricks.cluster.profile
>> - spark.databricks.io.cache.enabled
>> - spark.databricks.delta.optimizeWrite.enabled
>> - spark.databricks.passthrough.enabled
>>
>> Some users or developers may claim that there is a clear and huge
>> benefit. I must agree with them because it's also true.
>>
>> However, is this a way where Apache Spark project aims to go? I cannot
>> agree with that.
>>
>> It's very bad for Apache Spark distribution to support
>> `spark.databricks.*` namespace like Spark 3.5.4 because it's misleading to
>> the Apache Spark users by dilluting the boundary of Apache Spark
>> distribution (and Brand) and the commercial vendor products (and Brand).
>> Note that Apache Spark 3.5.5 and all future 3.5.x also support
>> `spark.databricks.*` until April 2025 (the end-of-life) because the
>> deprecation is not a deletion nor a ban.
>>
>> The incident at 3.5.4 was something that should never have happened. It
>> causes many confusions already and will more sadly.
>>
>> The confusion is contagious not only for the distribution, but also for
>> the source code. I guess
>> - The original Databricks contributor was confused what he contribute,
>> maybe.
>> - The Apache Spark committer (Jungtaek) overlooked what we should not
>> approve because the code
>>   resembles his internal company repo.
>> - The downstream Apache Spark fork repositories consume
>> `spark.databricks.*` namespace
>>   as Apache Spark's namespace.
>>
>> For me, it's more misleading to dillute the boundary of Apache Spark code
>> and the comercial vendor code.
>>
>> I have been working on this issue and considering this vote as the last
>> piece of the overall handling of the 'spark.databricks.*' incident because
>> I believe we are establishing a new rule for the Apache Spark community for
>> the future. This will serve as a precedent for handling similar incidents
>> in the future.
>>
>> Please let me re-summarize the past steps I did with the community:
>>
>> 1. Helping renaming the conf via SPARK-51172 (by approving it)
>>
>> 2. Banning `spark.databricks.*` via SPARK-51173 (by adding `configName`
>> Scalastyle rule)
>>
>> 3. Led the discussion thread
>> "Deprecating and banning `spark.databricks.*` config from Apache Spark
>> repository"
>> https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd
>>
>> 4. Reached the agreement to release Spark 3.5.5 early.
>> [VOTE] Release Apache Spark 3.5.5 deprecating `spark.databricks.*`
>> configuration
>> https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5
>>
>> 5. Releasing 3.5.5 as a release manager to provide a candidate migration
>> path
>>
>> 6. Proposing for 3.5.4 users to use 3.5.5 as the migration path
>>
>> I proposed to document on the migration guide of Spark 4.0 (Step 6)
>> because that is the only way to handle this incident without using the
>> specific vendor config name again in Spark code of `master` and
>> `branch-4.0` branch. As you read in Step 2 in the above, I prefer the
>> automatic way. The documentation-only solution never be my personal
>> preference. It was the lesser of two evils.
>>
>> Let me re-iterate this. Although we succeeded to deprecate the
>> configuration early, the contaminated release branch `branch-3.5` and its
>> releases still support the configuration to the Spark jobs until April 2026
>> (the end-of-life). It's a long-standing live incident which is happening
>> currently.
>>
>> For the vote, "Retain ... in Spark 4.0.x", I casted -1 because it aims to
>> introduce the vendor configuration name (string) back to Apache Spark 4
>> code. It means another contaminated `branch-4.0` will blur the boundary.
>>
>> On top of that, the Databricks' Apache Spark committer (Jungtaek), who
>> caused this incident by merging `spark.databricks.*` code, set a trap on
>> this vote by writing the following when he initiated the vote.
>>
>> > if someone supports including migration logic to be longer than Spark
>> 4.0.x, please cast +1 here and leave the desired last minor version of
>> Spark to retain this migration logic.
>>
>> At the same time, he casted +1 with the following.
>>
>> > Starting from my +1 (non-binding).
>> > In addition, I propose to retain migration logic till Spark 4.1.x and
>> remove it in Spark 4.2.0.
>>
>> In the open source community, he is playing his own card trick by
>> flipping the vote title under everyone's nose like a magic.
>>
>> > [VOTE] Retain migration logic of incorrect `spark.databricks.*` config
>> in Spark 4.0.x
>> > [VOTE] Retain migration logic of incorrect `spark.databricks.*` config
>> in Spark 4.0.x/4.1.x
>>
>> In other words, Jungtaek is trying to spread this terrible and misleading
>> situation to the end of life of Spark 4.1.0 (Spring 2027) for now. I can
>> say that he will extend it again by ignoring the removal at Spark 4.2+ with
>> the same reasons like the following.
>> - We usually don't introduce the breaking behavior under the same major
>> version.
>> - The maintenance cost is near zero.
>> In this case, it will be permanent under Spark 4 (~ 2030?)
>>
>> Of course, someone might say that it's better than `branch-3.5` situation
>> because the migration code is a read-only support. However, it's still in
>> the same category which misleads the community to the confusion where
>> Apache Spark supports `spark.databricks.*` configurations.
>>
>> The vote was articulated to cause a longer and bigger side-effect because
>> `branch-4.0` and `branch-4.1` has a longer period and many releases in
>> total. To prevent the breakout of contagious `spark.databricks.*`
>> situations, we should stop now and protect `branch-4.0`. The side-effect
>> and implication is huge.
>>
>> Apache Spark 4.0.0 is the only version we can stop this spreading. So,
>> documentation only is the only feasible way to choose.
>>
>> So, -1 (= The technical justification for the veto is valid)
>>
>> Sincerely,
>> Dongjoon.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Jungtaek Lim <ka...@gmail.com>]

Dongjoon,

I'm now OK with whatever you think, but I argue your vote is technically
moot since it's about your vote justification, and I have no binding vote
to counter you. Let's be fair.

On Sun, Mar 16, 2025 at 3:07 PM Dongjoon Hyun <do...@apache.org> wrote:

> Thank you for focusing on this, Mark.
>
> I also agree with you that this should be decided by the Apache Spark PMC
> and appreciate the effort to help us move forward in the Apache way.
>
> As you mentioned, there is no ASF policy. That's true.
>
> > I am not aware of any ASF policy that strictly forbids the mention of a
> vendor
> > in Apache code for any reason
>
> Let's imagine that Apache Spark project starts to support the following
> existing vendor `spark.databricks.*` configs to help Spark users migrate
> (or offload) from Databricks service to open source Spark cluster easily.
>
> - spark.databricks.cluster.profile
> - spark.databricks.io.cache.enabled
> - spark.databricks.delta.optimizeWrite.enabled
> - spark.databricks.passthrough.enabled
>
> Some users or developers may claim that there is a clear and huge benefit.
> I must agree with them because it's also true.
>
> However, is this a way where Apache Spark project aims to go? I cannot
> agree with that.
>
> It's very bad for Apache Spark distribution to support
> `spark.databricks.*` namespace like Spark 3.5.4 because it's misleading to
> the Apache Spark users by dilluting the boundary of Apache Spark
> distribution (and Brand) and the commercial vendor products (and Brand).
> Note that Apache Spark 3.5.5 and all future 3.5.x also support
> `spark.databricks.*` until April 2025 (the end-of-life) because the
> deprecation is not a deletion nor a ban.
>
> The incident at 3.5.4 was something that should never have happened. It
> causes many confusions already and will more sadly.
>
> The confusion is contagious not only for the distribution, but also for
> the source code. I guess
> - The original Databricks contributor was confused what he contribute,
> maybe.
> - The Apache Spark committer (Jungtaek) overlooked what we should not
> approve because the code
>   resembles his internal company repo.
> - The downstream Apache Spark fork repositories consume
> `spark.databricks.*` namespace
>   as Apache Spark's namespace.
>
> For me, it's more misleading to dillute the boundary of Apache Spark code
> and the comercial vendor code.
>
> I have been working on this issue and considering this vote as the last
> piece of the overall handling of the 'spark.databricks.*' incident because
> I believe we are establishing a new rule for the Apache Spark community for
> the future. This will serve as a precedent for handling similar incidents
> in the future.
>
> Please let me re-summarize the past steps I did with the community:
>
> 1. Helping renaming the conf via SPARK-51172 (by approving it)
>
> 2. Banning `spark.databricks.*` via SPARK-51173 (by adding `configName`
> Scalastyle rule)
>
> 3. Led the discussion thread
> "Deprecating and banning `spark.databricks.*` config from Apache Spark
> repository"
> https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd
>
> 4. Reached the agreement to release Spark 3.5.5 early.
> [VOTE] Release Apache Spark 3.5.5 deprecating `spark.databricks.*`
> configuration
> https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5
>
> 5. Releasing 3.5.5 as a release manager to provide a candidate migration
> path
>
> 6. Proposing for 3.5.4 users to use 3.5.5 as the migration path
>
> I proposed to document on the migration guide of Spark 4.0 (Step 6)
> because that is the only way to handle this incident without using the
> specific vendor config name again in Spark code of `master` and
> `branch-4.0` branch. As you read in Step 2 in the above, I prefer the
> automatic way. The documentation-only solution never be my personal
> preference. It was the lesser of two evils.
>
> Let me re-iterate this. Although we succeeded to deprecate the
> configuration early, the contaminated release branch `branch-3.5` and its
> releases still support the configuration to the Spark jobs until April 2026
> (the end-of-life). It's a long-standing live incident which is happening
> currently.
>
> For the vote, "Retain ... in Spark 4.0.x", I casted -1 because it aims to
> introduce the vendor configuration name (string) back to Apache Spark 4
> code. It means another contaminated `branch-4.0` will blur the boundary.
>
> On top of that, the Databricks' Apache Spark committer (Jungtaek), who
> caused this incident by merging `spark.databricks.*` code, set a trap on
> this vote by writing the following when he initiated the vote.
>
> > if someone supports including migration logic to be longer than Spark
> 4.0.x, please cast +1 here and leave the desired last minor version of
> Spark to retain this migration logic.
>
> At the same time, he casted +1 with the following.
>
> > Starting from my +1 (non-binding).
> > In addition, I propose to retain migration logic till Spark 4.1.x and
> remove it in Spark 4.2.0.
>
> In the open source community, he is playing his own card trick by flipping
> the vote title under everyone's nose like a magic.
>
> > [VOTE] Retain migration logic of incorrect `spark.databricks.*` config
> in Spark 4.0.x
> > [VOTE] Retain migration logic of incorrect `spark.databricks.*` config
> in Spark 4.0.x/4.1.x
>
> In other words, Jungtaek is trying to spread this terrible and misleading
> situation to the end of life of Spark 4.1.0 (Spring 2027) for now. I can
> say that he will extend it again by ignoring the removal at Spark 4.2+ with
> the same reasons like the following.
> - We usually don't introduce the breaking behavior under the same major
> version.
> - The maintenance cost is near zero.
> In this case, it will be permanent under Spark 4 (~ 2030?)
>
> Of course, someone might say that it's better than `branch-3.5` situation
> because the migration code is a read-only support. However, it's still in
> the same category which misleads the community to the confusion where
> Apache Spark supports `spark.databricks.*` configurations.
>
> The vote was articulated to cause a longer and bigger side-effect because
> `branch-4.0` and `branch-4.1` has a longer period and many releases in
> total. To prevent the breakout of contagious `spark.databricks.*`
> situations, we should stop now and protect `branch-4.0`. The side-effect
> and implication is huge.
>
> Apache Spark 4.0.0 is the only version we can stop this spreading. So,
> documentation only is the only feasible way to choose.
>
> So, -1 (= The technical justification for the veto is valid)
>
> Sincerely,
> Dongjoon.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

[Dongjoon Hyun <do...@apache.org>]

Thank you for focusing on this, Mark.

I also agree with you that this should be decided by the Apache Spark PMC and appreciate the effort to help us move forward in the Apache way.

As you mentioned, there is no ASF policy. That's true.

> I am not aware of any ASF policy that strictly forbids the mention of a vendor
> in Apache code for any reason

Let's imagine that Apache Spark project starts to support the following existing vendor `spark.databricks.*` configs to help Spark users migrate (or offload) from Databricks service to open source Spark cluster easily.

- spark.databricks.cluster.profile
- spark.databricks.io.cache.enabled
- spark.databricks.delta.optimizeWrite.enabled
- spark.databricks.passthrough.enabled

Some users or developers may claim that there is a clear and huge benefit. I must agree with them because it's also true.

However, is this a way where Apache Spark project aims to go? I cannot agree with that.

It's very bad for Apache Spark distribution to support `spark.databricks.*` namespace like Spark 3.5.4 because it's misleading to the Apache Spark users by dilluting the boundary of Apache Spark distribution (and Brand) and the commercial vendor products (and Brand). Note that Apache Spark 3.5.5 and all future 3.5.x also support `spark.databricks.*` until April 2025 (the end-of-life) because the deprecation is not a deletion nor a ban.

The incident at 3.5.4 was something that should never have happened. It causes many confusions already and will more sadly.

The confusion is contagious not only for the distribution, but also for the source code. I guess
- The original Databricks contributor was confused what he contribute, maybe.
- The Apache Spark committer (Jungtaek) overlooked what we should not approve because the code
  resembles his internal company repo.
- The downstream Apache Spark fork repositories consume `spark.databricks.*` namespace
  as Apache Spark's namespace.

For me, it's more misleading to dillute the boundary of Apache Spark code and the comercial vendor code.

I have been working on this issue and considering this vote as the last piece of the overall handling of the 'spark.databricks.*' incident because I believe we are establishing a new rule for the Apache Spark community for the future. This will serve as a precedent for handling similar incidents in the future.

Please let me re-summarize the past steps I did with the community:

1. Helping renaming the conf via SPARK-51172 (by approving it)

2. Banning `spark.databricks.*` via SPARK-51173 (by adding `configName` Scalastyle rule)

3. Led the discussion thread
"Deprecating and banning `spark.databricks.*` config from Apache Spark repository"
https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd

4. Reached the agreement to release Spark 3.5.5 early.
[VOTE] Release Apache Spark 3.5.5 deprecating `spark.databricks.*` configuration
https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5

5. Releasing 3.5.5 as a release manager to provide a candidate migration path

6. Proposing for 3.5.4 users to use 3.5.5 as the migration path

I proposed to document on the migration guide of Spark 4.0 (Step 6) because that is the only way to handle this incident without using the specific vendor config name again in Spark code of `master` and `branch-4.0` branch. As you read in Step 2 in the above, I prefer the automatic way. The documentation-only solution never be my personal preference. It was the lesser of two evils.

Let me re-iterate this. Although we succeeded to deprecate the configuration early, the contaminated release branch `branch-3.5` and its releases still support the configuration to the Spark jobs until April 2026 (the end-of-life). It's a long-standing live incident which is happening currently.

For the vote, "Retain ... in Spark 4.0.x", I casted -1 because it aims to introduce the vendor configuration name (string) back to Apache Spark 4 code. It means another contaminated `branch-4.0` will blur the boundary.

On top of that, the Databricks' Apache Spark committer (Jungtaek), who caused this incident by merging `spark.databricks.*` code, set a trap on this vote by writing the following when he initiated the vote.

> if someone supports including migration logic to be longer than Spark 4.0.x, please cast +1 here and leave the desired last minor version of Spark to retain this migration logic.

At the same time, he casted +1 with the following.

> Starting from my +1 (non-binding).
> In addition, I propose to retain migration logic till Spark 4.1.x and remove it in Spark 4.2.0.

In the open source community, he is playing his own card trick by flipping the vote title under everyone's nose like a magic.

> [VOTE] Retain migration logic of incorrect `spark.databricks.*` config in Spark 4.0.x
> [VOTE] Retain migration logic of incorrect `spark.databricks.*` config in Spark 4.0.x/4.1.x

In other words, Jungtaek is trying to spread this terrible and misleading situation to the end of life of Spark 4.1.0 (Spring 2027) for now. I can say that he will extend it again by ignoring the removal at Spark 4.2+ with the same reasons like the following.
- We usually don't introduce the breaking behavior under the same major version.
- The maintenance cost is near zero.
In this case, it will be permanent under Spark 4 (~ 2030?)

Of course, someone might say that it's better than `branch-3.5` situation because the migration code is a read-only support. However, it's still in the same category which misleads the community to the confusion where Apache Spark supports `spark.databricks.*` configurations.

The vote was articulated to cause a longer and bigger side-effect because `branch-4.0` and `branch-4.1` has a longer period and many releases in total. To prevent the breakout of contagious `spark.databricks.*` situations, we should stop now and protect `branch-4.0`. The side-effect and implication is huge.

Apache Spark 4.0.0 is the only version we can stop this spreading. So, documentation only is the only feasible way to choose.

So, -1 (= The technical justification for the veto is valid)

Sincerely,
Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org