You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Szehon Ho <sz...@gmail.com> on 2026/10/01 03:28:20 UTC

Re: [VOTE] Release Spark 4.2.1 (RC1)

I took a look at the recent 4.3 fixes/blockers.  We fixed two for 4.2, but
we decided not to do the other two for 4.2, as listed below.

   - SPARK-59804 <https://issues.apache.org/jira/browse/SPARK-59804> — PR
   #59074 <https://github.com/apache/spark/pull/59074> — *merged into 4.2*
   - SPARK-51830 <https://issues.apache.org/jira/browse/SPARK-51830> — PR
   #57994 <https://github.com/apache/spark/pull/57994>;    *merged into 4.2*
   - SPARK-44462 <https://issues.apache.org/jira/browse/SPARK-44462> — PR
   #55410 <https://github.com/apache/spark/pull/55410> — took a look.  it
   is a complex fix with some risk, and there's some behavior change.  We are
   leaning towards omitting it from 4.2.1.
   - SPARK-59122 <https://issues.apache.org/jira/browse/SPARK-59122> — PR
   #58419 <https://github.com/apache/spark/pull/58419> — took a look.  It
   is also not so simple and also UNION codegen is disabled on 4.2, so also
   leaning towards omitting it from 4.2.1.

I am not sure we need to wait until 4.3 RC passes, as the scope of 4.2.1
maintenance release should be a more focused fix for 4.2 users hitting
obvious issues (example, on Iceberg side).  As we see from some examples,
some 4.3 issues may not be appropriate for 4.2.  I would prefer to restart
the process and halt only if there is an unresolved 4.2 blocker, unless
someone feels strongly otherwise.

Thanks,
Szehon

On Mon, Sep 28, 2026 at 7:01 PM Holden Karau <ho...@gmail.com> wrote:

> I think it might be good to hold the point releases until the 4.3 RC
> passes just incase we’ve got any blockers we discover and want to backport?
> I don’t think it’s a “must” hold just it might make life easier.
>
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
> On Mon, Sep 28, 2026 at 6:20 PM Uroš Bojanić <ur...@apache.org> wrote:
>
>> Hi folks, thank you for working on this! Please also let me know if any
>> of the issues above are blockers for other releases (e.g. 4.1.4, 4.0.5) or
>> not; there are separate threads for those releases as well.
>>
>> On 2026/09/28 09:11:21 szehon@apache.org wrote:
>> > Please vote on releasing the following candidate as Apache Spark
>> version 4.2.1.
>> >
>> > The vote is open until Thu, 01 Oct 2026 03:11:20 PDT and passes if a
>> majority +1 PMC votes are cast, with
>> > a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 4.2.1
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see https://spark.apache.org/
>> >
>> > The tag to be voted on is v4.2.1-rc1 (commit 72c18760854):
>> > https://github.com/apache/spark/tree/v4.2.1-rc1
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://downloads.apache.org/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1531/
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-docs/
>> >
>> > The list of bug fixes going into 4.2.1 can be found at the following
>> URL:
>> > https://issues.apache.org/jira/projects/SPARK/versions/12357191
>> >
>> > FAQ
>> >
>> > =========================
>> > How can I help test this release?
>> > =========================
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC via "pip install
>> https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/pyspark-4.2.1.tar.gz
>> "
>> > and see if anything important breaks.
>> > In the Java/Scala, you can add the staging repository to your project's
>> resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with an out of date RC going forward).
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Release Spark 4.2.1 (RC1)

Posted by Szehon Ho <sz...@gmail.com>.
Officially failing the 4.2.0 RC1.

Also updating the list, https://github.com/apache/spark/pull/58929 is
merged (the last relevant 4.3.0 blocker that I know) , so will cut 4.2.1
RC2 shortly, please take a look when it's out.

Thanks everyone for voting and commenting.
Szehon

On Thu, Oct 1, 2026 at 6:08 AM Jungtaek Lim <ka...@gmail.com>
wrote:

> https://github.com/apache/spark/pull/58929
> If I understand correctly about the issue, this is a blocker for 4.2 as it
> can break checkpoint in stateful workloads (despite that v2 is not the
> default format). The issue isn't framed well enough to emphasize the
> severity; once we have a new JIRA ticket (or rewritten JIRA ticket), I'll
> bump the severity to blocker.
>
> On Thu, Oct 1, 2026 at 12:29 PM Szehon Ho <sz...@gmail.com> wrote:
>
>> I took a look at the recent 4.3 fixes/blockers.  We fixed two for 4.2,
>> but we decided not to do the other two for 4.2, as listed below.
>>
>>    - SPARK-59804 <https://issues.apache.org/jira/browse/SPARK-59804> — PR
>>    #59074 <https://github.com/apache/spark/pull/59074> — *merged into
>>    4.2*
>>    - SPARK-51830 <https://issues.apache.org/jira/browse/SPARK-51830> — PR
>>    #57994 <https://github.com/apache/spark/pull/57994>;    *merged into
>>    4.2*
>>    - SPARK-44462 <https://issues.apache.org/jira/browse/SPARK-44462> — PR
>>    #55410 <https://github.com/apache/spark/pull/55410> — took a
>>    look.  it is a complex fix with some risk, and there's some behavior
>>    change.  We are leaning towards omitting it from 4.2.1.
>>    - SPARK-59122 <https://issues.apache.org/jira/browse/SPARK-59122> — PR
>>    #58419 <https://github.com/apache/spark/pull/58419> — took a look.
>>    It is also not so simple and also UNION codegen is disabled on 4.2, so also
>>    leaning towards omitting it from 4.2.1.
>>
>> I am not sure we need to wait until 4.3 RC passes, as the scope of 4.2.1
>> maintenance release should be a more focused fix for 4.2 users hitting
>> obvious issues (example, on Iceberg side).  As we see from some examples,
>> some 4.3 issues may not be appropriate for 4.2.  I would prefer to restart
>> the process and halt only if there is an unresolved 4.2 blocker, unless
>> someone feels strongly otherwise.
>>
>> Thanks,
>> Szehon
>>
>> On Mon, Sep 28, 2026 at 7:01 PM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> I think it might be good to hold the point releases until the 4.3 RC
>>> passes just incase we’ve got any blockers we discover and want to backport?
>>> I don’t think it’s a “must” hold just it might make life easier.
>>>
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>> On Mon, Sep 28, 2026 at 6:20 PM Uroš Bojanić <ur...@apache.org> wrote:
>>>
>>>> Hi folks, thank you for working on this! Please also let me know if any
>>>> of the issues above are blockers for other releases (e.g. 4.1.4, 4.0.5) or
>>>> not; there are separate threads for those releases as well.
>>>>
>>>> On 2026/09/28 09:11:21 szehon@apache.org wrote:
>>>> > Please vote on releasing the following candidate as Apache Spark
>>>> version 4.2.1.
>>>> >
>>>> > The vote is open until Thu, 01 Oct 2026 03:11:20 PDT and passes if a
>>>> majority +1 PMC votes are cast, with
>>>> > a minimum of 3 +1 votes.
>>>> >
>>>> > [ ] +1 Release this package as Apache Spark 4.2.1
>>>> > [ ] -1 Do not release this package because ...
>>>> >
>>>> > To learn more about Apache Spark, please see
>>>> https://spark.apache.org/
>>>> >
>>>> > The tag to be voted on is v4.2.1-rc1 (commit 72c18760854):
>>>> > https://github.com/apache/spark/tree/v4.2.1-rc1
>>>> >
>>>> > The release files, including signatures, digests, etc. can be found
>>>> at:
>>>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/
>>>> >
>>>> > Signatures used for Spark RCs can be found in this file:
>>>> > https://downloads.apache.org/spark/KEYS
>>>> >
>>>> > The staging repository for this release can be found at:
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachespark-1531/
>>>> >
>>>> > The documentation corresponding to this release can be found at:
>>>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-docs/
>>>> >
>>>> > The list of bug fixes going into 4.2.1 can be found at the following
>>>> URL:
>>>> > https://issues.apache.org/jira/projects/SPARK/versions/12357191
>>>> >
>>>> > FAQ
>>>> >
>>>> > =========================
>>>> > How can I help test this release?
>>>> > =========================
>>>> >
>>>> > If you are a Spark user, you can help us test this release by taking
>>>> > an existing Spark workload and running on this release candidate, then
>>>> > reporting any regressions.
>>>> >
>>>> > If you're working in PySpark you can set up a virtual env and install
>>>> > the current RC via "pip install
>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/pyspark-4.2.1.tar.gz
>>>> "
>>>> > and see if anything important breaks.
>>>> > In the Java/Scala, you can add the staging repository to your
>>>> project's resolvers and test
>>>> > with the RC (make sure to clean up the artifact cache before/after so
>>>> > you don't end up building with an out of date RC going forward).
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>

Re: [VOTE] Release Spark 4.2.1 (RC1)

Posted by Jungtaek Lim <ka...@gmail.com>.
https://github.com/apache/spark/pull/58929
If I understand correctly about the issue, this is a blocker for 4.2 as it
can break checkpoint in stateful workloads (despite that v2 is not the
default format). The issue isn't framed well enough to emphasize the
severity; once we have a new JIRA ticket (or rewritten JIRA ticket), I'll
bump the severity to blocker.

On Thu, Oct 1, 2026 at 12:29 PM Szehon Ho <sz...@gmail.com> wrote:

> I took a look at the recent 4.3 fixes/blockers.  We fixed two for 4.2, but
> we decided not to do the other two for 4.2, as listed below.
>
>    - SPARK-59804 <https://issues.apache.org/jira/browse/SPARK-59804> — PR
>    #59074 <https://github.com/apache/spark/pull/59074> — *merged into 4.2*
>    - SPARK-51830 <https://issues.apache.org/jira/browse/SPARK-51830> — PR
>    #57994 <https://github.com/apache/spark/pull/57994>;    *merged into
>    4.2*
>    - SPARK-44462 <https://issues.apache.org/jira/browse/SPARK-44462> — PR
>    #55410 <https://github.com/apache/spark/pull/55410> — took a look.  it
>    is a complex fix with some risk, and there's some behavior change.  We are
>    leaning towards omitting it from 4.2.1.
>    - SPARK-59122 <https://issues.apache.org/jira/browse/SPARK-59122> — PR
>    #58419 <https://github.com/apache/spark/pull/58419> — took a look.  It
>    is also not so simple and also UNION codegen is disabled on 4.2, so also
>    leaning towards omitting it from 4.2.1.
>
> I am not sure we need to wait until 4.3 RC passes, as the scope of 4.2.1
> maintenance release should be a more focused fix for 4.2 users hitting
> obvious issues (example, on Iceberg side).  As we see from some examples,
> some 4.3 issues may not be appropriate for 4.2.  I would prefer to restart
> the process and halt only if there is an unresolved 4.2 blocker, unless
> someone feels strongly otherwise.
>
> Thanks,
> Szehon
>
> On Mon, Sep 28, 2026 at 7:01 PM Holden Karau <ho...@gmail.com>
> wrote:
>
>> I think it might be good to hold the point releases until the 4.3 RC
>> passes just incase we’ve got any blockers we discover and want to backport?
>> I don’t think it’s a “must” hold just it might make life easier.
>>
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>> On Mon, Sep 28, 2026 at 6:20 PM Uroš Bojanić <ur...@apache.org> wrote:
>>
>>> Hi folks, thank you for working on this! Please also let me know if any
>>> of the issues above are blockers for other releases (e.g. 4.1.4, 4.0.5) or
>>> not; there are separate threads for those releases as well.
>>>
>>> On 2026/09/28 09:11:21 szehon@apache.org wrote:
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 4.2.1.
>>> >
>>> > The vote is open until Thu, 01 Oct 2026 03:11:20 PDT and passes if a
>>> majority +1 PMC votes are cast, with
>>> > a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 4.2.1
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see https://spark.apache.org/
>>> >
>>> > The tag to be voted on is v4.2.1-rc1 (commit 72c18760854):
>>> > https://github.com/apache/spark/tree/v4.2.1-rc1
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://downloads.apache.org/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1531/
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-docs/
>>> >
>>> > The list of bug fixes going into 4.2.1 can be found at the following
>>> URL:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/12357191
>>> >
>>> > FAQ
>>> >
>>> > =========================
>>> > How can I help test this release?
>>> > =========================
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC via "pip install
>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/pyspark-4.2.1.tar.gz
>>> "
>>> > and see if anything important breaks.
>>> > In the Java/Scala, you can add the staging repository to your
>>> project's resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with an out of date RC going forward).
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>