You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jungtaek Lim <ka...@gmail.com> on 2026/09/28 22:51:49 UTC
Blockers for Apache Spark 4.3.0 RC
Hi dev, I'm a release manager of Apache Spark 4.3.0.
We failed the release candidate 1 for Apache Spark 4.3.0, and I'm trying to
find the timing to cut the next RC. Last time I checked, there are ongoing
blockers (myself also filed a blocker ticket) so I kept delaying to trigger
a new RC.
Since there were a couple weeks, I would like to re-check the blockers
before moving forward. Instead of just checking the JIRA tickets, I'd love
to collect the information from folks, who own blocker ticket(s) and the
rationale/justification of the blocker, and what is the estimation to
resolve it.
Would you folks mind helping me to obtain the information? This will be
very helpful to find the timing to cut RC2 rather than blindly cutting RC2
and seeing it fail.
Thanks!
Jungtaek Lim (HeartSaVioR)
Re: Blockers for Apache Spark 4.3.0 RC
Posted by Jungtaek Lim <ka...@gmail.com>.
Heads-up: We've addressed the blocker ticket
https://issues.apache.org/jira/browse/SPARK-59919. I'm waiting for
branch-4.3 CI (CI running against the latest commit) and will cut a new RC
once it's green.
On Thu, Oct 1, 2026 at 5:55 PM Jungtaek Lim <ka...@gmail.com>
wrote:
> Looks like tickets are addressed or just waiting for CI to merge. Thanks
> all!
>
> Btw I've found a blocker which impacts Spark 4.2.0+
> https://issues.apache.org/jira/browse/SPARK-59919
> I'll ensure the fix for this blocker is landed, and trigger another RC.
>
>
> On Wed, Sep 30, 2026 at 12:45 PM Jungtaek Lim <
> kabhwan.opensource@gmail.com> wrote:
>
>> Yes, we can wait for both tickets. Please try best to get it done and
>> consider excluding if it takes time (e.g. an week) and there is a
>> workaround.
>>
>> I've marked SPARK-44462 (https://github.com/apache/spark/pull/55410) as
>> a blocker (author is Jie Yang), but I've reviewed the PR and found a
>> couple blockers, so maybe we will have to leave it as known issue if those
>> are not addressed in the proper time.
>>
>> SPARK-59122 (https://github.com/apache/spark/pull/58419) is also marked
>> as a blocker (author is Jie Yang) - the PR got approved by Wenchen/Dongjoon
>> so I expect the PR to be merged sooner than later.
>>
>> I don't think I see other blocker tickets at this point.
>>
>> On Wed, Sep 30, 2026 at 1:38 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Similar to 4.1.4/4.2.1 RC1, we found that branch-4.3 also includes
>>> SPARK-59407 with two regressions: the byte-valued event-log limit is
>>> checked against twice the UTF-16 character count, and the public JVM
>>> no-argument constructor of ReplayListenerBus was removed.
>>>
>>> PR #59074 addresses these issues, along with line diagnostics and memory
>>> limits, and is still under review.
>>>
>>> Could we hold off cutting RC1 before merging the reviewed fix? This
>>> would avoid shipping the current configuration semantics and changing them
>>> in a subsequent patch release.
>>>
>>> https://github.com/apache/spark/pull/59074
>>>
>>>
>>> On Mon, Sep 28, 2026 at 5:25 PM Gengliang Wang <lt...@gmail.com> wrote:
>>>
>>>> Hi Jungtaek,
>>>>
>>>> I would like to get SPARK-59841
>>>> <https://issues.apache.org/jira/browse/SPARK-59841> (PR #59119
>>>> <https://github.com/apache/spark/pull/59119>) merged for Spark 4.3 to
>>>> address an inconsistency between the loadTable and loadChangelog APIs.
>>>>
>>>> Would it be possible to allow another 1–2 days for review and merge
>>>> before proceeding with the release?
>>>>
>>>> Thanks!
>>>>
>>>> On Mon, Sep 28, 2026 at 3:54 PM Holden Karau <ho...@gmail.com>
>>>> wrote:
>>>>
>>>>> I’m investigating a back backport I made during some fixes and hope to
>>>>> have that resolved tonight, but otherwise I think we’re looking good.
>>>>>
>>>>>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>> Pronouns: she/her
>>>>>
>>>>> On Mon, Sep 28, 2026 at 3:52 PM Jungtaek Lim <
>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>
>>>>>> Hi dev, I'm a release manager of Apache Spark 4.3.0.
>>>>>>
>>>>>> We failed the release candidate 1 for Apache Spark 4.3.0, and I'm
>>>>>> trying to find the timing to cut the next RC. Last time I checked, there
>>>>>> are ongoing blockers (myself also filed a blocker ticket) so I kept
>>>>>> delaying to trigger a new RC.
>>>>>>
>>>>>> Since there were a couple weeks, I would like to re-check the
>>>>>> blockers before moving forward. Instead of just checking the JIRA tickets,
>>>>>> I'd love to collect the information from folks, who own blocker ticket(s)
>>>>>> and the rationale/justification of the blocker, and what is the estimation
>>>>>> to resolve it.
>>>>>>
>>>>>> Would you folks mind helping me to obtain the information? This will
>>>>>> be very helpful to find the timing to cut RC2 rather than blindly cutting
>>>>>> RC2 and seeing it fail.
>>>>>>
>>>>>> Thanks!
>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>
>>>>>
Re: Blockers for Apache Spark 4.3.0 RC
Posted by Jungtaek Lim <ka...@gmail.com>.
Looks like tickets are addressed or just waiting for CI to merge. Thanks
all!
Btw I've found a blocker which impacts Spark 4.2.0+
https://issues.apache.org/jira/browse/SPARK-59919
I'll ensure the fix for this blocker is landed, and trigger another RC.
On Wed, Sep 30, 2026 at 12:45 PM Jungtaek Lim <ka...@gmail.com>
wrote:
> Yes, we can wait for both tickets. Please try best to get it done and
> consider excluding if it takes time (e.g. an week) and there is a
> workaround.
>
> I've marked SPARK-44462 (https://github.com/apache/spark/pull/55410) as a
> blocker (author is Jie Yang), but I've reviewed the PR and found a couple
> blockers, so maybe we will have to leave it as known issue if those are not
> addressed in the proper time.
>
> SPARK-59122 (https://github.com/apache/spark/pull/58419) is also marked
> as a blocker (author is Jie Yang) - the PR got approved by Wenchen/Dongjoon
> so I expect the PR to be merged sooner than later.
>
> I don't think I see other blocker tickets at this point.
>
> On Wed, Sep 30, 2026 at 1:38 AM L. C. Hsieh <vi...@gmail.com> wrote:
>
>> Hello,
>>
>> Similar to 4.1.4/4.2.1 RC1, we found that branch-4.3 also includes
>> SPARK-59407 with two regressions: the byte-valued event-log limit is
>> checked against twice the UTF-16 character count, and the public JVM
>> no-argument constructor of ReplayListenerBus was removed.
>>
>> PR #59074 addresses these issues, along with line diagnostics and memory
>> limits, and is still under review.
>>
>> Could we hold off cutting RC1 before merging the reviewed fix? This would
>> avoid shipping the current configuration semantics and changing them in a
>> subsequent patch release.
>>
>> https://github.com/apache/spark/pull/59074
>>
>>
>> On Mon, Sep 28, 2026 at 5:25 PM Gengliang Wang <lt...@gmail.com> wrote:
>>
>>> Hi Jungtaek,
>>>
>>> I would like to get SPARK-59841
>>> <https://issues.apache.org/jira/browse/SPARK-59841> (PR #59119
>>> <https://github.com/apache/spark/pull/59119>) merged for Spark 4.3 to
>>> address an inconsistency between the loadTable and loadChangelog APIs.
>>>
>>> Would it be possible to allow another 1–2 days for review and merge
>>> before proceeding with the release?
>>>
>>> Thanks!
>>>
>>> On Mon, Sep 28, 2026 at 3:54 PM Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> I’m investigating a back backport I made during some fixes and hope to
>>>> have that resolved tonight, but otherwise I think we’re looking good.
>>>>
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>> On Mon, Sep 28, 2026 at 3:52 PM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>>
>>>>> Hi dev, I'm a release manager of Apache Spark 4.3.0.
>>>>>
>>>>> We failed the release candidate 1 for Apache Spark 4.3.0, and I'm
>>>>> trying to find the timing to cut the next RC. Last time I checked, there
>>>>> are ongoing blockers (myself also filed a blocker ticket) so I kept
>>>>> delaying to trigger a new RC.
>>>>>
>>>>> Since there were a couple weeks, I would like to re-check the blockers
>>>>> before moving forward. Instead of just checking the JIRA tickets, I'd love
>>>>> to collect the information from folks, who own blocker ticket(s) and the
>>>>> rationale/justification of the blocker, and what is the estimation to
>>>>> resolve it.
>>>>>
>>>>> Would you folks mind helping me to obtain the information? This will
>>>>> be very helpful to find the timing to cut RC2 rather than blindly cutting
>>>>> RC2 and seeing it fail.
>>>>>
>>>>> Thanks!
>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>
Re: Blockers for Apache Spark 4.3.0 RC
Posted by Jungtaek Lim <ka...@gmail.com>.
Yes, we can wait for both tickets. Please try best to get it done and
consider excluding if it takes time (e.g. an week) and there is a
workaround.
I've marked SPARK-44462 (https://github.com/apache/spark/pull/55410) as a
blocker (author is Jie Yang), but I've reviewed the PR and found a couple
blockers, so maybe we will have to leave it as known issue if those are not
addressed in the proper time.
SPARK-59122 (https://github.com/apache/spark/pull/58419) is also marked as
a blocker (author is Jie Yang) - the PR got approved by Wenchen/Dongjoon so
I expect the PR to be merged sooner than later.
I don't think I see other blocker tickets at this point.
On Wed, Sep 30, 2026 at 1:38 AM L. C. Hsieh <vi...@gmail.com> wrote:
> Hello,
>
> Similar to 4.1.4/4.2.1 RC1, we found that branch-4.3 also includes
> SPARK-59407 with two regressions: the byte-valued event-log limit is
> checked against twice the UTF-16 character count, and the public JVM
> no-argument constructor of ReplayListenerBus was removed.
>
> PR #59074 addresses these issues, along with line diagnostics and memory
> limits, and is still under review.
>
> Could we hold off cutting RC1 before merging the reviewed fix? This would
> avoid shipping the current configuration semantics and changing them in a
> subsequent patch release.
>
> https://github.com/apache/spark/pull/59074
>
>
> On Mon, Sep 28, 2026 at 5:25 PM Gengliang Wang <lt...@gmail.com> wrote:
>
>> Hi Jungtaek,
>>
>> I would like to get SPARK-59841
>> <https://issues.apache.org/jira/browse/SPARK-59841> (PR #59119
>> <https://github.com/apache/spark/pull/59119>) merged for Spark 4.3 to
>> address an inconsistency between the loadTable and loadChangelog APIs.
>>
>> Would it be possible to allow another 1–2 days for review and merge
>> before proceeding with the release?
>>
>> Thanks!
>>
>> On Mon, Sep 28, 2026 at 3:54 PM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> I’m investigating a back backport I made during some fixes and hope to
>>> have that resolved tonight, but otherwise I think we’re looking good.
>>>
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>> On Mon, Sep 28, 2026 at 3:52 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> Hi dev, I'm a release manager of Apache Spark 4.3.0.
>>>>
>>>> We failed the release candidate 1 for Apache Spark 4.3.0, and I'm
>>>> trying to find the timing to cut the next RC. Last time I checked, there
>>>> are ongoing blockers (myself also filed a blocker ticket) so I kept
>>>> delaying to trigger a new RC.
>>>>
>>>> Since there were a couple weeks, I would like to re-check the blockers
>>>> before moving forward. Instead of just checking the JIRA tickets, I'd love
>>>> to collect the information from folks, who own blocker ticket(s) and the
>>>> rationale/justification of the blocker, and what is the estimation to
>>>> resolve it.
>>>>
>>>> Would you folks mind helping me to obtain the information? This will be
>>>> very helpful to find the timing to cut RC2 rather than blindly cutting RC2
>>>> and seeing it fail.
>>>>
>>>> Thanks!
>>>> Jungtaek Lim (HeartSaVioR)
>>>>
>>>
Re: Blockers for Apache Spark 4.3.0 RC
Posted by "L. C. Hsieh" <vi...@gmail.com>.
Hello,
Similar to 4.1.4/4.2.1 RC1, we found that branch-4.3 also includes
SPARK-59407 with two regressions: the byte-valued event-log limit is
checked against twice the UTF-16 character count, and the public JVM
no-argument constructor of ReplayListenerBus was removed.
PR #59074 addresses these issues, along with line diagnostics and memory
limits, and is still under review.
Could we hold off cutting RC1 before merging the reviewed fix? This would
avoid shipping the current configuration semantics and changing them in a
subsequent patch release.
https://github.com/apache/spark/pull/59074
On Mon, Sep 28, 2026 at 5:25 PM Gengliang Wang <lt...@gmail.com> wrote:
> Hi Jungtaek,
>
> I would like to get SPARK-59841
> <https://issues.apache.org/jira/browse/SPARK-59841> (PR #59119
> <https://github.com/apache/spark/pull/59119>) merged for Spark 4.3 to
> address an inconsistency between the loadTable and loadChangelog APIs.
>
> Would it be possible to allow another 1–2 days for review and merge before
> proceeding with the release?
>
> Thanks!
>
> On Mon, Sep 28, 2026 at 3:54 PM Holden Karau <ho...@gmail.com>
> wrote:
>
>> I’m investigating a back backport I made during some fixes and hope to
>> have that resolved tonight, but otherwise I think we’re looking good.
>>
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>> On Mon, Sep 28, 2026 at 3:52 PM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> Hi dev, I'm a release manager of Apache Spark 4.3.0.
>>>
>>> We failed the release candidate 1 for Apache Spark 4.3.0, and I'm trying
>>> to find the timing to cut the next RC. Last time I checked, there are
>>> ongoing blockers (myself also filed a blocker ticket) so I kept delaying to
>>> trigger a new RC.
>>>
>>> Since there were a couple weeks, I would like to re-check the blockers
>>> before moving forward. Instead of just checking the JIRA tickets, I'd love
>>> to collect the information from folks, who own blocker ticket(s) and the
>>> rationale/justification of the blocker, and what is the estimation to
>>> resolve it.
>>>
>>> Would you folks mind helping me to obtain the information? This will be
>>> very helpful to find the timing to cut RC2 rather than blindly cutting RC2
>>> and seeing it fail.
>>>
>>> Thanks!
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>
Re: Blockers for Apache Spark 4.3.0 RC
Posted by Gengliang Wang <lt...@gmail.com>.
Hi Jungtaek,
I would like to get SPARK-59841
<https://issues.apache.org/jira/browse/SPARK-59841> (PR #59119
<https://github.com/apache/spark/pull/59119>) merged for Spark 4.3 to
address an inconsistency between the loadTable and loadChangelog APIs.
Would it be possible to allow another 1–2 days for review and merge before
proceeding with the release?
Thanks!
On Mon, Sep 28, 2026 at 3:54 PM Holden Karau <ho...@gmail.com> wrote:
> I’m investigating a back backport I made during some fixes and hope to
> have that resolved tonight, but otherwise I think we’re looking good.
>
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
> On Mon, Sep 28, 2026 at 3:52 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Hi dev, I'm a release manager of Apache Spark 4.3.0.
>>
>> We failed the release candidate 1 for Apache Spark 4.3.0, and I'm trying
>> to find the timing to cut the next RC. Last time I checked, there are
>> ongoing blockers (myself also filed a blocker ticket) so I kept delaying to
>> trigger a new RC.
>>
>> Since there were a couple weeks, I would like to re-check the blockers
>> before moving forward. Instead of just checking the JIRA tickets, I'd love
>> to collect the information from folks, who own blocker ticket(s) and the
>> rationale/justification of the blocker, and what is the estimation to
>> resolve it.
>>
>> Would you folks mind helping me to obtain the information? This will be
>> very helpful to find the timing to cut RC2 rather than blindly cutting RC2
>> and seeing it fail.
>>
>> Thanks!
>> Jungtaek Lim (HeartSaVioR)
>>
>
Re: Blockers for Apache Spark 4.3.0 RC
Posted by Holden Karau <ho...@gmail.com>.
I’m investigating a back backport I made during some fixes and hope to have
that resolved tonight, but otherwise I think we’re looking good.
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her
On Mon, Sep 28, 2026 at 3:52 PM Jungtaek Lim <ka...@gmail.com>
wrote:
> Hi dev, I'm a release manager of Apache Spark 4.3.0.
>
> We failed the release candidate 1 for Apache Spark 4.3.0, and I'm trying
> to find the timing to cut the next RC. Last time I checked, there are
> ongoing blockers (myself also filed a blocker ticket) so I kept delaying to
> trigger a new RC.
>
> Since there were a couple weeks, I would like to re-check the blockers
> before moving forward. Instead of just checking the JIRA tickets, I'd love
> to collect the information from folks, who own blocker ticket(s) and the
> rationale/justification of the blocker, and what is the estimation to
> resolve it.
>
> Would you folks mind helping me to obtain the information? This will be
> very helpful to find the timing to cut RC2 rather than blindly cutting RC2
> and seeing it fail.
>
> Thanks!
> Jungtaek Lim (HeartSaVioR)
>