You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dongjoon Hyun <do...@apache.org> on 2025/07/01 21:17:49 UTC
Re: [DISCUSS] SPIP: Monthly preview release
Thank you so much for the suggestion and achieving the automated infra, Hyujkin.
I have a few questions.
1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") as "Q8. Final Success" criteria. I'm wondering if you have some schedule in your mind for Spark 5.0 in next 2 years?
2. Are we going to lower the bar for the monthly preview releases? Specifically, I'm wondering if the Preview-RC1 supposed to pass always because it's a preview release? As we know, it's not until now. For example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK 4.0.0-preview1 (RC3)".
3. Is SPIP proposing monthly previews for only the FIRST MINOR versions like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? There is no `4.0.1-preview1`?
4. Although there was an automated test email for 3.5.7, SPIP is not aiming maintenance version release like 4.0.1 and 3.5.7?
Initially, I thought you were going to propose (4) "Automated Monthly Maintenance Release".
For me, (4) is more beneficial than this SPIP because
- We spend the same community effort during voting (including at least 3 PMC votes) for (4) and SPIP.
- (4) has the real benefit because users can use it in the production while SPIP didn't.
Technically, it could be a little weird if Apache Spark community releases only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the actual maintenance versions like `4.0.1`, ..., `4.0.2`.
In short, "Automated Monthly Maintenance Release" might be the prerequisite for "Monthly Preview Release". What do you think about that? Can we extend your SPIP in this direction?
Thanks,
Dongjoon.
On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
> Hi all,
>
> I would like to propose a monthly preview for our dev branch, e.g., Spark
> 4.1.0 preview1 ... previewN.
>
> Per https://issues.apache.org/jira/browse/SPARK-52176, we have minimized
> the manual work so I think it's realistic to propose this.
>
> Couple of notes:
> - The manual steps it requires would be to run GitHub Actions twice for RC
> and publishing, and summarizing the vote result. There IS a way to even
> automate this but it needs more work to comply with ASF policy. I would
> like to stick to this minimal manual work for now.
> - For now, I would like to volunteer to be responsible for the preview
> releases and incrementally improve our release policy guidelines (
> https://spark.apache.org/release-process.html) as well once this SPIP
> passes.
> - The individual release would be, I suspect, about the first week in each
> month but I would like to avoid setting the explicit date in the SPIP so it
> makes us less pressured.
>
> JIRA: https://issues.apache.org/jira/browse/SPARK-52625
> SPIP:
> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Jungtaek Lim <ka...@gmail.com>.
IMHO it's probably better to avoid trying to figure out a workaround if
that is the "policy" of ASF.
Btw, I guess the main point here is, what is the definition of "acceptable"
release. For example, do we feel overloaded to just verify the signature
per month? I think we wouldn't. The reason the topic of overhead on the
voting came up is because we are not very clear about what the shape of
release we want to have for previews.
IIUC about SPIP (also answers from Hyukjin w.r.t. my questions), the
demand of having preview releases is closer to the demand we want to "tag"
the snapshot at some point and release artifacts so that people (especially
early adopters and 3rd parties) do not need to struggle with how to pull
snapshot artifacts, or even have to build on their own. If that is the case
and we are happy with that direction, maybe the fact CI passes would be
enough to give +1 to the release along with signature verification.
If we want to provide specific meaning to the preview release (e.g. we
release preview for feature A being completed), it's probably non-trivial
to set the timeline in prior and it has to be flexible rather than specific
interval. But if we are on the same page that it's really a pain to play
with a snapshot and we want to provide some checkpointed artifacts in the
middle of development, I think this is worth giving a try. At least it is
not a one way door and interval is not a strict one which can change later
upon more lightweight discussion.
On Thu, Jul 3, 2025 at 8:59 AM Hyukjin Kwon <gu...@apache.org> wrote:
> Great point. I would eventually like to remove the vote overhead as well.
> It is required per ASF policy but I am actually taking a look around to
> work around this.
> I took it as monthly because it seems to be pretty enough given my
> experience of releasing Spark 3.5.6 via automated GitHub Actions.
>
> It's more to try it out and see how it goes actually so I think we won't
> have to worry too much.
> I also don't want us to strictly force the monthly release - we could skip
> a couple if there's anything happening.
> If things don't go well, we can always change / adjust timelines, etc.
>
>
> On Wed, 2 Jul 2025 at 20:39, Mark Hamstra <ma...@gmail.com> wrote:
>
>> On Tue, Jul 1, 2025 at 9:42 PM Hyukjin Kwon <gu...@apache.org> wrote:
>> >
>> > > 1. Since we "release" the preview, we will go through the VOTE
>> process. What is the expected overhead on doing this monthly? (Maybe this
>> would be coupled with the questions below.)
>> >
>> > We have to vote for every preview every month for now. We can try and
>> see how it goes.
>>
>> The voting requirement does make me question the overhead as well. I'm
>> not sure that we can expect the number of PMC members necessary for a
>> valid vote to give a meaningful release review every month. That would
>> leave us with automated preview releases with invalid or failed
>> release votes, or perhaps even worse, preview-releases that have been
>> rubber-stamped without meaningful review. Beyond naming/accounting
>> details such as whether the next preview release after preview-n fails
>> to pass the vote threshold is a potentially confusing repeat of
>> preview-n or automatically becomes preview-n+1 or preview-n+0.1 or
>> something, a build up of failed, non-released "releases" would seem to
>> create its own uncertainty, friction and overhead.
>>
>> Given arguendo that the span between actual releases is currently too
>> long and that the other extreme of an automated release after every
>> commit would be too short, what is the basis for monthly instead of
>> maybe something like bi-monthly or quarterly being the right
>> preview-release cadence?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Hyukjin Kwon <gu...@apache.org>.
Great point. I would eventually like to remove the vote overhead as well.
It is required per ASF policy but I am actually taking a look around to
work around this.
I took it as monthly because it seems to be pretty enough given my
experience of releasing Spark 3.5.6 via automated GitHub Actions.
It's more to try it out and see how it goes actually so I think we won't
have to worry too much.
I also don't want us to strictly force the monthly release - we could skip
a couple if there's anything happening.
If things don't go well, we can always change / adjust timelines, etc.
On Wed, 2 Jul 2025 at 20:39, Mark Hamstra <ma...@gmail.com> wrote:
> On Tue, Jul 1, 2025 at 9:42 PM Hyukjin Kwon <gu...@apache.org> wrote:
> >
> > > 1. Since we "release" the preview, we will go through the VOTE
> process. What is the expected overhead on doing this monthly? (Maybe this
> would be coupled with the questions below.)
> >
> > We have to vote for every preview every month for now. We can try and
> see how it goes.
>
> The voting requirement does make me question the overhead as well. I'm
> not sure that we can expect the number of PMC members necessary for a
> valid vote to give a meaningful release review every month. That would
> leave us with automated preview releases with invalid or failed
> release votes, or perhaps even worse, preview-releases that have been
> rubber-stamped without meaningful review. Beyond naming/accounting
> details such as whether the next preview release after preview-n fails
> to pass the vote threshold is a potentially confusing repeat of
> preview-n or automatically becomes preview-n+1 or preview-n+0.1 or
> something, a build up of failed, non-released "releases" would seem to
> create its own uncertainty, friction and overhead.
>
> Given arguendo that the span between actual releases is currently too
> long and that the other extreme of an automated release after every
> commit would be too short, what is the basis for monthly instead of
> maybe something like bi-monthly or quarterly being the right
> preview-release cadence?
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Mark Hamstra <ma...@gmail.com>.
On Tue, Jul 1, 2025 at 9:42 PM Hyukjin Kwon <gu...@apache.org> wrote:
>
> > 1. Since we "release" the preview, we will go through the VOTE process. What is the expected overhead on doing this monthly? (Maybe this would be coupled with the questions below.)
>
> We have to vote for every preview every month for now. We can try and see how it goes.
The voting requirement does make me question the overhead as well. I'm
not sure that we can expect the number of PMC members necessary for a
valid vote to give a meaningful release review every month. That would
leave us with automated preview releases with invalid or failed
release votes, or perhaps even worse, preview-releases that have been
rubber-stamped without meaningful review. Beyond naming/accounting
details such as whether the next preview release after preview-n fails
to pass the vote threshold is a potentially confusing repeat of
preview-n or automatically becomes preview-n+1 or preview-n+0.1 or
something, a build up of failed, non-released "releases" would seem to
create its own uncertainty, friction and overhead.
Given arguendo that the span between actual releases is currently too
long and that the other extreme of an automated release after every
commit would be too short, what is the basis for monthly instead of
maybe something like bi-monthly or quarterly being the right
preview-release cadence?
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Jungtaek Lim <ka...@gmail.com>.
Sounds good. I think this is worth giving a try and we can adjust upon
lessons learned if any. Thanks for the proposal!
On Wed, Jul 2, 2025 at 1:42 PM Hyukjin Kwon <gu...@apache.org> wrote:
> > 1. Since we "release" the preview, we will go through the VOTE process.
> What is the expected overhead on doing this monthly? (Maybe this would be
> coupled with the questions below.)
>
> We have to vote for every preview every month for now. We can try and see
> how it goes.
>
> > 2. What quality do we foresee on the preview release? Do we expect to
> have -1 and address them per monthly release, if there is an unresolved
> correctness issue or regression? Or will we have very loose criteria of
> verification and leave the preview to early adopters with risks? Is the
> stability of preview release based on the fact existing unit tests are
> passing, and we are mostly verifying the signatures of artifacts?
>
> I actually expect almost no -1 unless the release is completely useless
> (e.g., can't run anything or can't run Spark submission). But I would
> expect the community to decide this :-).
>
> > 3. Do we allow breaking changes across previews, say, as long as we
> don't introduce breaking changes in the new minor version it should be fine?
>
> Yeah. It's not an official release. It should be fine to break.
>
> > 4. Does this drive the change of our development
> in other directions e.g. dev branch or so? e.g. transformWithState had been
> marked as private at the phase of development since we knew it would take a
> lot of time. Or will we allow preview releases to contain incomplete
> features? How do we prevent users from accessing the incomplete feature, or
> do we intend to try it out while it's still in development?
>
> We won't apply any compatibility policies here I believe. Should be
> treated as the same as our past preview releases which does not guarantee
> anything.
>
>
> On Wed, 2 Jul 2025 at 11:54, Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Thanks for the proposal. The direction is awesome - we have such a long
>> interval between minor releases and this would help us address some of the
>> issues from the long release cadence.
>>
>> I'd like to understand a couple of things.
>>
>> 1. Since we "release" the preview, we will go through the VOTE process.
>> What is the expected overhead on doing this monthly? (Maybe this would be
>> coupled with the questions below.)
>>
>> 2. What quality do we foresee on the preview release? Do we expect to
>> have -1 and address them per monthly release, if there is an unresolved
>> correctness issue or regression? Or will we have very loose criteria of
>> verification and leave the preview to early adopters with risks? Is the
>> stability of preview release based on the fact existing unit tests are
>> passing, and we are mostly verifying the signatures of artifacts?
>>
>> 3. Do we allow breaking changes across previews, say, as long as we don't
>> introduce breaking changes in the new minor version it should be fine?
>>
>> 4. Does this drive the change of our development in other directions e.g.
>> dev branch or so? e.g. transformWithState had been marked as private at the
>> phase of development since we knew it would take a lot of time. Or will we
>> allow preview releases to contain incomplete features? How do we prevent
>> users from accessing the incomplete feature, or do we intend to try it out
>> while it's still in development?
>>
>>
>>
>> On Wed, Jul 2, 2025 at 11:25 AM Hyukjin Kwon <gu...@apache.org>
>> wrote:
>>
>>> Let me start the vote tmr if we're all good with this :-).
>>>
>>> On Wed, 2 Jul 2025 at 10:12, Anton Okolnychyi <ao...@gmail.com>
>>> wrote:
>>>
>>>> Having monthly preview releases for Spark is going to be huge for
>>>> projects like Iceberg and Delta.
>>>>
>>>> - Anton
>>>>
>>>> On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Thank you for the clarification, Hyukjin. Also, thank you for sharing
>>>>> your direction, DB.
>>>>>
>>>>> I agree with you folks that the AS-IS scope of SPIP is a good start.
>>>>>
>>>>> +1 for the SPIP because `4.1.0-previewX` itself is actually very
>>>>> helpful already during developing Spark subprojects like "Spark Connect for
>>>>> Swift" and "Spark K8s Operator". :-)
>>>>>
>>>>> Thank you again.
>>>>>
>>>>> Dongjoon.
>>>>>
>>>>> On 2025/07/02 00:31:10 Hyukjin Kwon wrote:
>>>>> > Hi Dongjoon,
>>>>> >
>>>>> > Thanks a lot for your detailed feedback and great questions!
>>>>> > Let me clarify my current proposal and thoughts:
>>>>> >
>>>>> > 1. Regarding Spark 5.0 schedule
>>>>> > At the moment, I don’t have a concrete Spark 5.0 schedule in mind.
>>>>> > I included the stable major releases in the Final Success criteria
>>>>> mainly
>>>>> > to set a practical milestone to complete the automation work and
>>>>> fully
>>>>> > transition to automated official releases.
>>>>> > I don't intend to set the next major release timeline in this SPIP.
>>>>> >
>>>>> > 2. Lowering the bar for preview releases
>>>>> > In short, yeah. I expect the bar for preview releases to be lower
>>>>> compared
>>>>> > to official releases, given that these previews are primarily for
>>>>> early
>>>>> > testing and feedback.
>>>>> > That said, if the community raises concerns during the vote and we
>>>>> end up
>>>>> > with multiple RCs, that’s totally fine. In such cases, we could even
>>>>> skip
>>>>> > the next month's preview if needed.
>>>>> > My intention is not to strictly enforce monthly previews but to
>>>>> provide
>>>>> > regular opportunities for testing, while keeping the process
>>>>> low-pressure
>>>>> > for the community.
>>>>> >
>>>>> > 3. Scope of monthly previews (first minor versions only)
>>>>> > Yup. This proposal is only for previews of the next minor version
>>>>> from the
>>>>> > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ...,
>>>>> until we
>>>>> > cut the real 4.1.0 release.
>>>>> > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on.
>>>>> > There will be no 4.0.1-preview1 style releases under this proposal.
>>>>> >
>>>>> > 4. Official releases (e.g., 4.0.1, 3.5.7)
>>>>> > For now, this SPIP does not target automating or introducing monthly
>>>>> > maintenance releases like 4.0.1.
>>>>> > But yeah, that's my final goal actually. The automated maintenance
>>>>> releases
>>>>> > are where I want to go next, after proving the automation works
>>>>> reliably
>>>>> > via previews.
>>>>> > here are actually some more work to be done to make it actually no
>>>>> manual
>>>>> > step at all.
>>>>> >
>>>>> >
>>>>> > *TL;DR*: this is a step before automating the official releases
>>>>> (it's not
>>>>> > tied to the official releases yet to be conservative) + providing
>>>>> users
>>>>> > with early access to the latest dev Spark build.
>>>>> >
>>>>> > On Wed, 2 Jul 2025 at 09:27, DB Tsai <db...@dbtsai.com> wrote:
>>>>> >
>>>>> > > Thank you, Hyukjin for driving the SPIP and for your work on the
>>>>> release
>>>>> > > automation infrastructure — it’s a huge step forward.
>>>>> > >
>>>>> > > I’ve been thinking about this topic quite a bit since the Spark 4.0
>>>>> > > release. While Spark continues to deliver meaningful improvements
>>>>> in every
>>>>> > > release and enjoys active community contributions, there’s a
>>>>> lingering
>>>>> > > perception that the project is mature but not evolving quickly. I
>>>>> feel this
>>>>> > > perception is largely due to the long gap between major versions —
>>>>> it’s
>>>>> > > been five years between Spark 3.0 and 4.0 — which has
>>>>> understandably caused
>>>>> > > some frustration among both contributors and users.
>>>>> > >
>>>>> > > Now, with release automation and monthly preview builds purposed
>>>>> in this
>>>>> > > SPIP, we have a real opportunity to change that. As Dongjoon
>>>>> suggested,
>>>>> > > setting up a regular maintenance release cadence — perhaps
>>>>> bi-monthly
>>>>> > > instead of monthly — could strike the right balance and make these
>>>>> builds
>>>>> > > more viable for production environments.
>>>>> > >
>>>>> > > If this model proves successful, we could move toward an even
>>>>> faster major
>>>>> > > release cadence and designate one LTS versions annually, with
>>>>> extended
>>>>> > > backport support.
>>>>> > >
>>>>> > > Benefits for the OSS Community:
>>>>> > >
>>>>> > > - Faster time-to-production for new features
>>>>> > > - Stronger contributor engagement
>>>>> > > - Quicker community feedback cycles
>>>>> > > - Easier debugging and testing through smaller, incremental
>>>>> changes
>>>>> > >
>>>>> > >
>>>>> > > Thanks,
>>>>> > >
>>>>> > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>>>> > >
>>>>> > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org>
>>>>> wrote:
>>>>> > >
>>>>> > > Thank you so much for the suggestion and achieving the automated
>>>>> infra,
>>>>> > > Hyujkin.
>>>>> > >
>>>>> > > I have a few questions.
>>>>> > >
>>>>> > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major
>>>>> releases") as
>>>>> > > "Q8. Final Success" criteria. I'm wondering if you have some
>>>>> schedule in
>>>>> > > your mind for Spark 5.0 in next 2 years?
>>>>> > >
>>>>> > > 2. Are we going to lower the bar for the monthly preview releases?
>>>>> > > Specifically, I'm wondering if the Preview-RC1 supposed to pass
>>>>> always
>>>>> > > because it's a preview release? As we know, it's not until now. For
>>>>> > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK
>>>>> > > 4.0.0-preview1 (RC3)".
>>>>> > >
>>>>> > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR
>>>>> versions
>>>>> > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1?
>>>>> There is
>>>>> > > no `4.0.1-preview1`?
>>>>> > >
>>>>> > > 4. Although there was an automated test email for 3.5.7, SPIP is
>>>>> not
>>>>> > > aiming maintenance version release like 4.0.1 and 3.5.7?
>>>>> > >
>>>>> > > Initially, I thought you were going to propose (4) "Automated
>>>>> Monthly
>>>>> > > Maintenance Release".
>>>>> > >
>>>>> > > For me, (4) is more beneficial than this SPIP because
>>>>> > > - We spend the same community effort during voting (including at
>>>>> least 3
>>>>> > > PMC votes) for (4) and SPIP.
>>>>> > > - (4) has the real benefit because users can use it in the
>>>>> production
>>>>> > > while SPIP didn't.
>>>>> > >
>>>>> > > Technically, it could be a little weird if Apache Spark community
>>>>> releases
>>>>> > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering
>>>>> the actual
>>>>> > > maintenance versions like `4.0.1`, ..., `4.0.2`.
>>>>> > >
>>>>> > > In short, "Automated Monthly Maintenance Release" might be the
>>>>> > > prerequisite for "Monthly Preview Release". What do you think
>>>>> about that?
>>>>> > > Can we extend your SPIP in this direction?
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Dongjoon.
>>>>> > >
>>>>> > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
>>>>> > >
>>>>> > > Hi all,
>>>>> > >
>>>>> > > I would like to propose a monthly preview for our dev branch,
>>>>> e.g., Spark
>>>>> > > 4.1.0 preview1 ... previewN.
>>>>> > >
>>>>> > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have
>>>>> minimized
>>>>> > > the manual work so I think it's realistic to propose this.
>>>>> > >
>>>>> > > Couple of notes:
>>>>> > > - The manual steps it requires would be to run GitHub Actions
>>>>> twice for RC
>>>>> > > and publishing, and summarizing the vote result. There IS a way to
>>>>> even
>>>>> > > automate this but it needs more work to comply with ASF policy. I
>>>>> would
>>>>> > > like to stick to this minimal manual work for now.
>>>>> > > - For now, I would like to volunteer to be responsible for the
>>>>> preview
>>>>> > > releases and incrementally improve our release policy guidelines (
>>>>> > > https://spark.apache.org/release-process.html) as well once this
>>>>> SPIP
>>>>> > > passes.
>>>>> > > - The individual release would be, I suspect, about the first week
>>>>> in each
>>>>> > > month but I would like to avoid setting the explicit date in the
>>>>> SPIP so it
>>>>> > > makes us less pressured.
>>>>> > >
>>>>> > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625
>>>>> > > SPIP:
>>>>> > >
>>>>> > >
>>>>> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> ---------------------------------------------------------------------
>>>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Hyukjin Kwon <gu...@apache.org>.
> 1. Since we "release" the preview, we will go through the VOTE process.
What is the expected overhead on doing this monthly? (Maybe this would be
coupled with the questions below.)
We have to vote for every preview every month for now. We can try and see
how it goes.
> 2. What quality do we foresee on the preview release? Do we expect to
have -1 and address them per monthly release, if there is an unresolved
correctness issue or regression? Or will we have very loose criteria of
verification and leave the preview to early adopters with risks? Is the
stability of preview release based on the fact existing unit tests are
passing, and we are mostly verifying the signatures of artifacts?
I actually expect almost no -1 unless the release is completely useless
(e.g., can't run anything or can't run Spark submission). But I would
expect the community to decide this :-).
> 3. Do we allow breaking changes across previews, say, as long as we don't
introduce breaking changes in the new minor version it should be fine?
Yeah. It's not an official release. It should be fine to break.
> 4. Does this drive the change of our development in other directions e.g.
dev branch or so? e.g. transformWithState had been marked as private at the
phase of development since we knew it would take a lot of time. Or will we
allow preview releases to contain incomplete features? How do we prevent
users from accessing the incomplete feature, or do we intend to try it out
while it's still in development?
We won't apply any compatibility policies here I believe. Should be treated
as the same as our past preview releases which does not guarantee anything.
On Wed, 2 Jul 2025 at 11:54, Jungtaek Lim <ka...@gmail.com>
wrote:
> Thanks for the proposal. The direction is awesome - we have such a long
> interval between minor releases and this would help us address some of the
> issues from the long release cadence.
>
> I'd like to understand a couple of things.
>
> 1. Since we "release" the preview, we will go through the VOTE process.
> What is the expected overhead on doing this monthly? (Maybe this would be
> coupled with the questions below.)
>
> 2. What quality do we foresee on the preview release? Do we expect to have
> -1 and address them per monthly release, if there is an unresolved
> correctness issue or regression? Or will we have very loose criteria of
> verification and leave the preview to early adopters with risks? Is the
> stability of preview release based on the fact existing unit tests are
> passing, and we are mostly verifying the signatures of artifacts?
>
> 3. Do we allow breaking changes across previews, say, as long as we don't
> introduce breaking changes in the new minor version it should be fine?
>
> 4. Does this drive the change of our development in other directions e.g.
> dev branch or so? e.g. transformWithState had been marked as private at the
> phase of development since we knew it would take a lot of time. Or will we
> allow preview releases to contain incomplete features? How do we prevent
> users from accessing the incomplete feature, or do we intend to try it out
> while it's still in development?
>
>
>
> On Wed, Jul 2, 2025 at 11:25 AM Hyukjin Kwon <gu...@apache.org> wrote:
>
>> Let me start the vote tmr if we're all good with this :-).
>>
>> On Wed, 2 Jul 2025 at 10:12, Anton Okolnychyi <ao...@gmail.com>
>> wrote:
>>
>>> Having monthly preview releases for Spark is going to be huge for
>>> projects like Iceberg and Delta.
>>>
>>> - Anton
>>>
>>> On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> Thank you for the clarification, Hyukjin. Also, thank you for sharing
>>>> your direction, DB.
>>>>
>>>> I agree with you folks that the AS-IS scope of SPIP is a good start.
>>>>
>>>> +1 for the SPIP because `4.1.0-previewX` itself is actually very
>>>> helpful already during developing Spark subprojects like "Spark Connect for
>>>> Swift" and "Spark K8s Operator". :-)
>>>>
>>>> Thank you again.
>>>>
>>>> Dongjoon.
>>>>
>>>> On 2025/07/02 00:31:10 Hyukjin Kwon wrote:
>>>> > Hi Dongjoon,
>>>> >
>>>> > Thanks a lot for your detailed feedback and great questions!
>>>> > Let me clarify my current proposal and thoughts:
>>>> >
>>>> > 1. Regarding Spark 5.0 schedule
>>>> > At the moment, I don’t have a concrete Spark 5.0 schedule in mind.
>>>> > I included the stable major releases in the Final Success criteria
>>>> mainly
>>>> > to set a practical milestone to complete the automation work and fully
>>>> > transition to automated official releases.
>>>> > I don't intend to set the next major release timeline in this SPIP.
>>>> >
>>>> > 2. Lowering the bar for preview releases
>>>> > In short, yeah. I expect the bar for preview releases to be lower
>>>> compared
>>>> > to official releases, given that these previews are primarily for
>>>> early
>>>> > testing and feedback.
>>>> > That said, if the community raises concerns during the vote and we
>>>> end up
>>>> > with multiple RCs, that’s totally fine. In such cases, we could even
>>>> skip
>>>> > the next month's preview if needed.
>>>> > My intention is not to strictly enforce monthly previews but to
>>>> provide
>>>> > regular opportunities for testing, while keeping the process
>>>> low-pressure
>>>> > for the community.
>>>> >
>>>> > 3. Scope of monthly previews (first minor versions only)
>>>> > Yup. This proposal is only for previews of the next minor version
>>>> from the
>>>> > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ...,
>>>> until we
>>>> > cut the real 4.1.0 release.
>>>> > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on.
>>>> > There will be no 4.0.1-preview1 style releases under this proposal.
>>>> >
>>>> > 4. Official releases (e.g., 4.0.1, 3.5.7)
>>>> > For now, this SPIP does not target automating or introducing monthly
>>>> > maintenance releases like 4.0.1.
>>>> > But yeah, that's my final goal actually. The automated maintenance
>>>> releases
>>>> > are where I want to go next, after proving the automation works
>>>> reliably
>>>> > via previews.
>>>> > here are actually some more work to be done to make it actually no
>>>> manual
>>>> > step at all.
>>>> >
>>>> >
>>>> > *TL;DR*: this is a step before automating the official releases (it's
>>>> not
>>>> > tied to the official releases yet to be conservative) + providing
>>>> users
>>>> > with early access to the latest dev Spark build.
>>>> >
>>>> > On Wed, 2 Jul 2025 at 09:27, DB Tsai <db...@dbtsai.com> wrote:
>>>> >
>>>> > > Thank you, Hyukjin for driving the SPIP and for your work on the
>>>> release
>>>> > > automation infrastructure — it’s a huge step forward.
>>>> > >
>>>> > > I’ve been thinking about this topic quite a bit since the Spark 4.0
>>>> > > release. While Spark continues to deliver meaningful improvements
>>>> in every
>>>> > > release and enjoys active community contributions, there’s a
>>>> lingering
>>>> > > perception that the project is mature but not evolving quickly. I
>>>> feel this
>>>> > > perception is largely due to the long gap between major versions —
>>>> it’s
>>>> > > been five years between Spark 3.0 and 4.0 — which has
>>>> understandably caused
>>>> > > some frustration among both contributors and users.
>>>> > >
>>>> > > Now, with release automation and monthly preview builds purposed in
>>>> this
>>>> > > SPIP, we have a real opportunity to change that. As Dongjoon
>>>> suggested,
>>>> > > setting up a regular maintenance release cadence — perhaps
>>>> bi-monthly
>>>> > > instead of monthly — could strike the right balance and make these
>>>> builds
>>>> > > more viable for production environments.
>>>> > >
>>>> > > If this model proves successful, we could move toward an even
>>>> faster major
>>>> > > release cadence and designate one LTS versions annually, with
>>>> extended
>>>> > > backport support.
>>>> > >
>>>> > > Benefits for the OSS Community:
>>>> > >
>>>> > > - Faster time-to-production for new features
>>>> > > - Stronger contributor engagement
>>>> > > - Quicker community feedback cycles
>>>> > > - Easier debugging and testing through smaller, incremental
>>>> changes
>>>> > >
>>>> > >
>>>> > > Thanks,
>>>> > >
>>>> > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>>> > >
>>>> > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>> > >
>>>> > > Thank you so much for the suggestion and achieving the automated
>>>> infra,
>>>> > > Hyujkin.
>>>> > >
>>>> > > I have a few questions.
>>>> > >
>>>> > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major
>>>> releases") as
>>>> > > "Q8. Final Success" criteria. I'm wondering if you have some
>>>> schedule in
>>>> > > your mind for Spark 5.0 in next 2 years?
>>>> > >
>>>> > > 2. Are we going to lower the bar for the monthly preview releases?
>>>> > > Specifically, I'm wondering if the Preview-RC1 supposed to pass
>>>> always
>>>> > > because it's a preview release? As we know, it's not until now. For
>>>> > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK
>>>> > > 4.0.0-preview1 (RC3)".
>>>> > >
>>>> > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR
>>>> versions
>>>> > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1?
>>>> There is
>>>> > > no `4.0.1-preview1`?
>>>> > >
>>>> > > 4. Although there was an automated test email for 3.5.7, SPIP is not
>>>> > > aiming maintenance version release like 4.0.1 and 3.5.7?
>>>> > >
>>>> > > Initially, I thought you were going to propose (4) "Automated
>>>> Monthly
>>>> > > Maintenance Release".
>>>> > >
>>>> > > For me, (4) is more beneficial than this SPIP because
>>>> > > - We spend the same community effort during voting (including at
>>>> least 3
>>>> > > PMC votes) for (4) and SPIP.
>>>> > > - (4) has the real benefit because users can use it in the
>>>> production
>>>> > > while SPIP didn't.
>>>> > >
>>>> > > Technically, it could be a little weird if Apache Spark community
>>>> releases
>>>> > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the
>>>> actual
>>>> > > maintenance versions like `4.0.1`, ..., `4.0.2`.
>>>> > >
>>>> > > In short, "Automated Monthly Maintenance Release" might be the
>>>> > > prerequisite for "Monthly Preview Release". What do you think about
>>>> that?
>>>> > > Can we extend your SPIP in this direction?
>>>> > >
>>>> > > Thanks,
>>>> > > Dongjoon.
>>>> > >
>>>> > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
>>>> > >
>>>> > > Hi all,
>>>> > >
>>>> > > I would like to propose a monthly preview for our dev branch, e.g.,
>>>> Spark
>>>> > > 4.1.0 preview1 ... previewN.
>>>> > >
>>>> > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have
>>>> minimized
>>>> > > the manual work so I think it's realistic to propose this.
>>>> > >
>>>> > > Couple of notes:
>>>> > > - The manual steps it requires would be to run GitHub Actions twice
>>>> for RC
>>>> > > and publishing, and summarizing the vote result. There IS a way to
>>>> even
>>>> > > automate this but it needs more work to comply with ASF policy. I
>>>> would
>>>> > > like to stick to this minimal manual work for now.
>>>> > > - For now, I would like to volunteer to be responsible for the
>>>> preview
>>>> > > releases and incrementally improve our release policy guidelines (
>>>> > > https://spark.apache.org/release-process.html) as well once this
>>>> SPIP
>>>> > > passes.
>>>> > > - The individual release would be, I suspect, about the first week
>>>> in each
>>>> > > month but I would like to avoid setting the explicit date in the
>>>> SPIP so it
>>>> > > makes us less pressured.
>>>> > >
>>>> > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625
>>>> > > SPIP:
>>>> > >
>>>> > >
>>>> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>>>> > >
>>>> > >
>>>> > >
>>>> ---------------------------------------------------------------------
>>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >
>>>> > >
>>>> > >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks for the proposal. The direction is awesome - we have such a long
interval between minor releases and this would help us address some of the
issues from the long release cadence.
I'd like to understand a couple of things.
1. Since we "release" the preview, we will go through the VOTE process.
What is the expected overhead on doing this monthly? (Maybe this would be
coupled with the questions below.)
2. What quality do we foresee on the preview release? Do we expect to have
-1 and address them per monthly release, if there is an unresolved
correctness issue or regression? Or will we have very loose criteria of
verification and leave the preview to early adopters with risks? Is the
stability of preview release based on the fact existing unit tests are
passing, and we are mostly verifying the signatures of artifacts?
3. Do we allow breaking changes across previews, say, as long as we don't
introduce breaking changes in the new minor version it should be fine?
4. Does this drive the change of our development in other directions e.g.
dev branch or so? e.g. transformWithState had been marked as private at the
phase of development since we knew it would take a lot of time. Or will we
allow preview releases to contain incomplete features? How do we prevent
users from accessing the incomplete feature, or do we intend to try it out
while it's still in development?
On Wed, Jul 2, 2025 at 11:25 AM Hyukjin Kwon <gu...@apache.org> wrote:
> Let me start the vote tmr if we're all good with this :-).
>
> On Wed, 2 Jul 2025 at 10:12, Anton Okolnychyi <ao...@gmail.com>
> wrote:
>
>> Having monthly preview releases for Spark is going to be huge for
>> projects like Iceberg and Delta.
>>
>> - Anton
>>
>> On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <do...@apache.org> wrote:
>>
>>> Thank you for the clarification, Hyukjin. Also, thank you for sharing
>>> your direction, DB.
>>>
>>> I agree with you folks that the AS-IS scope of SPIP is a good start.
>>>
>>> +1 for the SPIP because `4.1.0-previewX` itself is actually very helpful
>>> already during developing Spark subprojects like "Spark Connect for Swift"
>>> and "Spark K8s Operator". :-)
>>>
>>> Thank you again.
>>>
>>> Dongjoon.
>>>
>>> On 2025/07/02 00:31:10 Hyukjin Kwon wrote:
>>> > Hi Dongjoon,
>>> >
>>> > Thanks a lot for your detailed feedback and great questions!
>>> > Let me clarify my current proposal and thoughts:
>>> >
>>> > 1. Regarding Spark 5.0 schedule
>>> > At the moment, I don’t have a concrete Spark 5.0 schedule in mind.
>>> > I included the stable major releases in the Final Success criteria
>>> mainly
>>> > to set a practical milestone to complete the automation work and fully
>>> > transition to automated official releases.
>>> > I don't intend to set the next major release timeline in this SPIP.
>>> >
>>> > 2. Lowering the bar for preview releases
>>> > In short, yeah. I expect the bar for preview releases to be lower
>>> compared
>>> > to official releases, given that these previews are primarily for early
>>> > testing and feedback.
>>> > That said, if the community raises concerns during the vote and we end
>>> up
>>> > with multiple RCs, that’s totally fine. In such cases, we could even
>>> skip
>>> > the next month's preview if needed.
>>> > My intention is not to strictly enforce monthly previews but to provide
>>> > regular opportunities for testing, while keeping the process
>>> low-pressure
>>> > for the community.
>>> >
>>> > 3. Scope of monthly previews (first minor versions only)
>>> > Yup. This proposal is only for previews of the next minor version from
>>> the
>>> > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until
>>> we
>>> > cut the real 4.1.0 release.
>>> > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on.
>>> > There will be no 4.0.1-preview1 style releases under this proposal.
>>> >
>>> > 4. Official releases (e.g., 4.0.1, 3.5.7)
>>> > For now, this SPIP does not target automating or introducing monthly
>>> > maintenance releases like 4.0.1.
>>> > But yeah, that's my final goal actually. The automated maintenance
>>> releases
>>> > are where I want to go next, after proving the automation works
>>> reliably
>>> > via previews.
>>> > here are actually some more work to be done to make it actually no
>>> manual
>>> > step at all.
>>> >
>>> >
>>> > *TL;DR*: this is a step before automating the official releases (it's
>>> not
>>> > tied to the official releases yet to be conservative) + providing users
>>> > with early access to the latest dev Spark build.
>>> >
>>> > On Wed, 2 Jul 2025 at 09:27, DB Tsai <db...@dbtsai.com> wrote:
>>> >
>>> > > Thank you, Hyukjin for driving the SPIP and for your work on the
>>> release
>>> > > automation infrastructure — it’s a huge step forward.
>>> > >
>>> > > I’ve been thinking about this topic quite a bit since the Spark 4.0
>>> > > release. While Spark continues to deliver meaningful improvements in
>>> every
>>> > > release and enjoys active community contributions, there’s a
>>> lingering
>>> > > perception that the project is mature but not evolving quickly. I
>>> feel this
>>> > > perception is largely due to the long gap between major versions —
>>> it’s
>>> > > been five years between Spark 3.0 and 4.0 — which has understandably
>>> caused
>>> > > some frustration among both contributors and users.
>>> > >
>>> > > Now, with release automation and monthly preview builds purposed in
>>> this
>>> > > SPIP, we have a real opportunity to change that. As Dongjoon
>>> suggested,
>>> > > setting up a regular maintenance release cadence — perhaps bi-monthly
>>> > > instead of monthly — could strike the right balance and make these
>>> builds
>>> > > more viable for production environments.
>>> > >
>>> > > If this model proves successful, we could move toward an even faster
>>> major
>>> > > release cadence and designate one LTS versions annually, with
>>> extended
>>> > > backport support.
>>> > >
>>> > > Benefits for the OSS Community:
>>> > >
>>> > > - Faster time-to-production for new features
>>> > > - Stronger contributor engagement
>>> > > - Quicker community feedback cycles
>>> > > - Easier debugging and testing through smaller, incremental
>>> changes
>>> > >
>>> > >
>>> > > Thanks,
>>> > >
>>> > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>> > >
>>> > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>> > >
>>> > > Thank you so much for the suggestion and achieving the automated
>>> infra,
>>> > > Hyujkin.
>>> > >
>>> > > I have a few questions.
>>> > >
>>> > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major
>>> releases") as
>>> > > "Q8. Final Success" criteria. I'm wondering if you have some
>>> schedule in
>>> > > your mind for Spark 5.0 in next 2 years?
>>> > >
>>> > > 2. Are we going to lower the bar for the monthly preview releases?
>>> > > Specifically, I'm wondering if the Preview-RC1 supposed to pass
>>> always
>>> > > because it's a preview release? As we know, it's not until now. For
>>> > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK
>>> > > 4.0.0-preview1 (RC3)".
>>> > >
>>> > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR
>>> versions
>>> > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1?
>>> There is
>>> > > no `4.0.1-preview1`?
>>> > >
>>> > > 4. Although there was an automated test email for 3.5.7, SPIP is not
>>> > > aiming maintenance version release like 4.0.1 and 3.5.7?
>>> > >
>>> > > Initially, I thought you were going to propose (4) "Automated Monthly
>>> > > Maintenance Release".
>>> > >
>>> > > For me, (4) is more beneficial than this SPIP because
>>> > > - We spend the same community effort during voting (including at
>>> least 3
>>> > > PMC votes) for (4) and SPIP.
>>> > > - (4) has the real benefit because users can use it in the production
>>> > > while SPIP didn't.
>>> > >
>>> > > Technically, it could be a little weird if Apache Spark community
>>> releases
>>> > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the
>>> actual
>>> > > maintenance versions like `4.0.1`, ..., `4.0.2`.
>>> > >
>>> > > In short, "Automated Monthly Maintenance Release" might be the
>>> > > prerequisite for "Monthly Preview Release". What do you think about
>>> that?
>>> > > Can we extend your SPIP in this direction?
>>> > >
>>> > > Thanks,
>>> > > Dongjoon.
>>> > >
>>> > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
>>> > >
>>> > > Hi all,
>>> > >
>>> > > I would like to propose a monthly preview for our dev branch, e.g.,
>>> Spark
>>> > > 4.1.0 preview1 ... previewN.
>>> > >
>>> > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have
>>> minimized
>>> > > the manual work so I think it's realistic to propose this.
>>> > >
>>> > > Couple of notes:
>>> > > - The manual steps it requires would be to run GitHub Actions twice
>>> for RC
>>> > > and publishing, and summarizing the vote result. There IS a way to
>>> even
>>> > > automate this but it needs more work to comply with ASF policy. I
>>> would
>>> > > like to stick to this minimal manual work for now.
>>> > > - For now, I would like to volunteer to be responsible for the
>>> preview
>>> > > releases and incrementally improve our release policy guidelines (
>>> > > https://spark.apache.org/release-process.html) as well once this
>>> SPIP
>>> > > passes.
>>> > > - The individual release would be, I suspect, about the first week
>>> in each
>>> > > month but I would like to avoid setting the explicit date in the
>>> SPIP so it
>>> > > makes us less pressured.
>>> > >
>>> > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625
>>> > > SPIP:
>>> > >
>>> > >
>>> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>>> > >
>>> > >
>>> > > ---------------------------------------------------------------------
>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Hyukjin Kwon <gu...@apache.org>.
Let me start the vote tmr if we're all good with this :-).
On Wed, 2 Jul 2025 at 10:12, Anton Okolnychyi <ao...@gmail.com> wrote:
> Having monthly preview releases for Spark is going to be huge for projects
> like Iceberg and Delta.
>
> - Anton
>
> On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> Thank you for the clarification, Hyukjin. Also, thank you for sharing
>> your direction, DB.
>>
>> I agree with you folks that the AS-IS scope of SPIP is a good start.
>>
>> +1 for the SPIP because `4.1.0-previewX` itself is actually very helpful
>> already during developing Spark subprojects like "Spark Connect for Swift"
>> and "Spark K8s Operator". :-)
>>
>> Thank you again.
>>
>> Dongjoon.
>>
>> On 2025/07/02 00:31:10 Hyukjin Kwon wrote:
>> > Hi Dongjoon,
>> >
>> > Thanks a lot for your detailed feedback and great questions!
>> > Let me clarify my current proposal and thoughts:
>> >
>> > 1. Regarding Spark 5.0 schedule
>> > At the moment, I don’t have a concrete Spark 5.0 schedule in mind.
>> > I included the stable major releases in the Final Success criteria
>> mainly
>> > to set a practical milestone to complete the automation work and fully
>> > transition to automated official releases.
>> > I don't intend to set the next major release timeline in this SPIP.
>> >
>> > 2. Lowering the bar for preview releases
>> > In short, yeah. I expect the bar for preview releases to be lower
>> compared
>> > to official releases, given that these previews are primarily for early
>> > testing and feedback.
>> > That said, if the community raises concerns during the vote and we end
>> up
>> > with multiple RCs, that’s totally fine. In such cases, we could even
>> skip
>> > the next month's preview if needed.
>> > My intention is not to strictly enforce monthly previews but to provide
>> > regular opportunities for testing, while keeping the process
>> low-pressure
>> > for the community.
>> >
>> > 3. Scope of monthly previews (first minor versions only)
>> > Yup. This proposal is only for previews of the next minor version from
>> the
>> > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until
>> we
>> > cut the real 4.1.0 release.
>> > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on.
>> > There will be no 4.0.1-preview1 style releases under this proposal.
>> >
>> > 4. Official releases (e.g., 4.0.1, 3.5.7)
>> > For now, this SPIP does not target automating or introducing monthly
>> > maintenance releases like 4.0.1.
>> > But yeah, that's my final goal actually. The automated maintenance
>> releases
>> > are where I want to go next, after proving the automation works reliably
>> > via previews.
>> > here are actually some more work to be done to make it actually no
>> manual
>> > step at all.
>> >
>> >
>> > *TL;DR*: this is a step before automating the official releases (it's
>> not
>> > tied to the official releases yet to be conservative) + providing users
>> > with early access to the latest dev Spark build.
>> >
>> > On Wed, 2 Jul 2025 at 09:27, DB Tsai <db...@dbtsai.com> wrote:
>> >
>> > > Thank you, Hyukjin for driving the SPIP and for your work on the
>> release
>> > > automation infrastructure — it’s a huge step forward.
>> > >
>> > > I’ve been thinking about this topic quite a bit since the Spark 4.0
>> > > release. While Spark continues to deliver meaningful improvements in
>> every
>> > > release and enjoys active community contributions, there’s a lingering
>> > > perception that the project is mature but not evolving quickly. I
>> feel this
>> > > perception is largely due to the long gap between major versions —
>> it’s
>> > > been five years between Spark 3.0 and 4.0 — which has understandably
>> caused
>> > > some frustration among both contributors and users.
>> > >
>> > > Now, with release automation and monthly preview builds purposed in
>> this
>> > > SPIP, we have a real opportunity to change that. As Dongjoon
>> suggested,
>> > > setting up a regular maintenance release cadence — perhaps bi-monthly
>> > > instead of monthly — could strike the right balance and make these
>> builds
>> > > more viable for production environments.
>> > >
>> > > If this model proves successful, we could move toward an even faster
>> major
>> > > release cadence and designate one LTS versions annually, with extended
>> > > backport support.
>> > >
>> > > Benefits for the OSS Community:
>> > >
>> > > - Faster time-to-production for new features
>> > > - Stronger contributor engagement
>> > > - Quicker community feedback cycles
>> > > - Easier debugging and testing through smaller, incremental changes
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>> > >
>> > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org>
>> wrote:
>> > >
>> > > Thank you so much for the suggestion and achieving the automated
>> infra,
>> > > Hyujkin.
>> > >
>> > > I have a few questions.
>> > >
>> > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases")
>> as
>> > > "Q8. Final Success" criteria. I'm wondering if you have some schedule
>> in
>> > > your mind for Spark 5.0 in next 2 years?
>> > >
>> > > 2. Are we going to lower the bar for the monthly preview releases?
>> > > Specifically, I'm wondering if the Preview-RC1 supposed to pass always
>> > > because it's a preview release? As we know, it's not until now. For
>> > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK
>> > > 4.0.0-preview1 (RC3)".
>> > >
>> > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR
>> versions
>> > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1?
>> There is
>> > > no `4.0.1-preview1`?
>> > >
>> > > 4. Although there was an automated test email for 3.5.7, SPIP is not
>> > > aiming maintenance version release like 4.0.1 and 3.5.7?
>> > >
>> > > Initially, I thought you were going to propose (4) "Automated Monthly
>> > > Maintenance Release".
>> > >
>> > > For me, (4) is more beneficial than this SPIP because
>> > > - We spend the same community effort during voting (including at
>> least 3
>> > > PMC votes) for (4) and SPIP.
>> > > - (4) has the real benefit because users can use it in the production
>> > > while SPIP didn't.
>> > >
>> > > Technically, it could be a little weird if Apache Spark community
>> releases
>> > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the
>> actual
>> > > maintenance versions like `4.0.1`, ..., `4.0.2`.
>> > >
>> > > In short, "Automated Monthly Maintenance Release" might be the
>> > > prerequisite for "Monthly Preview Release". What do you think about
>> that?
>> > > Can we extend your SPIP in this direction?
>> > >
>> > > Thanks,
>> > > Dongjoon.
>> > >
>> > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
>> > >
>> > > Hi all,
>> > >
>> > > I would like to propose a monthly preview for our dev branch, e.g.,
>> Spark
>> > > 4.1.0 preview1 ... previewN.
>> > >
>> > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have
>> minimized
>> > > the manual work so I think it's realistic to propose this.
>> > >
>> > > Couple of notes:
>> > > - The manual steps it requires would be to run GitHub Actions twice
>> for RC
>> > > and publishing, and summarizing the vote result. There IS a way to
>> even
>> > > automate this but it needs more work to comply with ASF policy. I
>> would
>> > > like to stick to this minimal manual work for now.
>> > > - For now, I would like to volunteer to be responsible for the preview
>> > > releases and incrementally improve our release policy guidelines (
>> > > https://spark.apache.org/release-process.html) as well once this SPIP
>> > > passes.
>> > > - The individual release would be, I suspect, about the first week in
>> each
>> > > month but I would like to avoid setting the explicit date in the SPIP
>> so it
>> > > makes us less pressured.
>> > >
>> > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625
>> > > SPIP:
>> > >
>> > >
>> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> > >
>> > >
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Anton Okolnychyi <ao...@gmail.com>.
Having monthly preview releases for Spark is going to be huge for projects
like Iceberg and Delta.
- Anton
On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <do...@apache.org> wrote:
> Thank you for the clarification, Hyukjin. Also, thank you for sharing your
> direction, DB.
>
> I agree with you folks that the AS-IS scope of SPIP is a good start.
>
> +1 for the SPIP because `4.1.0-previewX` itself is actually very helpful
> already during developing Spark subprojects like "Spark Connect for Swift"
> and "Spark K8s Operator". :-)
>
> Thank you again.
>
> Dongjoon.
>
> On 2025/07/02 00:31:10 Hyukjin Kwon wrote:
> > Hi Dongjoon,
> >
> > Thanks a lot for your detailed feedback and great questions!
> > Let me clarify my current proposal and thoughts:
> >
> > 1. Regarding Spark 5.0 schedule
> > At the moment, I don’t have a concrete Spark 5.0 schedule in mind.
> > I included the stable major releases in the Final Success criteria mainly
> > to set a practical milestone to complete the automation work and fully
> > transition to automated official releases.
> > I don't intend to set the next major release timeline in this SPIP.
> >
> > 2. Lowering the bar for preview releases
> > In short, yeah. I expect the bar for preview releases to be lower
> compared
> > to official releases, given that these previews are primarily for early
> > testing and feedback.
> > That said, if the community raises concerns during the vote and we end up
> > with multiple RCs, that’s totally fine. In such cases, we could even skip
> > the next month's preview if needed.
> > My intention is not to strictly enforce monthly previews but to provide
> > regular opportunities for testing, while keeping the process low-pressure
> > for the community.
> >
> > 3. Scope of monthly previews (first minor versions only)
> > Yup. This proposal is only for previews of the next minor version from
> the
> > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until we
> > cut the real 4.1.0 release.
> > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on.
> > There will be no 4.0.1-preview1 style releases under this proposal.
> >
> > 4. Official releases (e.g., 4.0.1, 3.5.7)
> > For now, this SPIP does not target automating or introducing monthly
> > maintenance releases like 4.0.1.
> > But yeah, that's my final goal actually. The automated maintenance
> releases
> > are where I want to go next, after proving the automation works reliably
> > via previews.
> > here are actually some more work to be done to make it actually no manual
> > step at all.
> >
> >
> > *TL;DR*: this is a step before automating the official releases (it's not
> > tied to the official releases yet to be conservative) + providing users
> > with early access to the latest dev Spark build.
> >
> > On Wed, 2 Jul 2025 at 09:27, DB Tsai <db...@dbtsai.com> wrote:
> >
> > > Thank you, Hyukjin for driving the SPIP and for your work on the
> release
> > > automation infrastructure — it’s a huge step forward.
> > >
> > > I’ve been thinking about this topic quite a bit since the Spark 4.0
> > > release. While Spark continues to deliver meaningful improvements in
> every
> > > release and enjoys active community contributions, there’s a lingering
> > > perception that the project is mature but not evolving quickly. I feel
> this
> > > perception is largely due to the long gap between major versions — it’s
> > > been five years between Spark 3.0 and 4.0 — which has understandably
> caused
> > > some frustration among both contributors and users.
> > >
> > > Now, with release automation and monthly preview builds purposed in
> this
> > > SPIP, we have a real opportunity to change that. As Dongjoon suggested,
> > > setting up a regular maintenance release cadence — perhaps bi-monthly
> > > instead of monthly — could strike the right balance and make these
> builds
> > > more viable for production environments.
> > >
> > > If this model proves successful, we could move toward an even faster
> major
> > > release cadence and designate one LTS versions annually, with extended
> > > backport support.
> > >
> > > Benefits for the OSS Community:
> > >
> > > - Faster time-to-production for new features
> > > - Stronger contributor engagement
> > > - Quicker community feedback cycles
> > > - Easier debugging and testing through smaller, incremental changes
> > >
> > >
> > > Thanks,
> > >
> > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> > >
> > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org> wrote:
> > >
> > > Thank you so much for the suggestion and achieving the automated infra,
> > > Hyujkin.
> > >
> > > I have a few questions.
> > >
> > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases")
> as
> > > "Q8. Final Success" criteria. I'm wondering if you have some schedule
> in
> > > your mind for Spark 5.0 in next 2 years?
> > >
> > > 2. Are we going to lower the bar for the monthly preview releases?
> > > Specifically, I'm wondering if the Preview-RC1 supposed to pass always
> > > because it's a preview release? As we know, it's not until now. For
> > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK
> > > 4.0.0-preview1 (RC3)".
> > >
> > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR versions
> > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1?
> There is
> > > no `4.0.1-preview1`?
> > >
> > > 4. Although there was an automated test email for 3.5.7, SPIP is not
> > > aiming maintenance version release like 4.0.1 and 3.5.7?
> > >
> > > Initially, I thought you were going to propose (4) "Automated Monthly
> > > Maintenance Release".
> > >
> > > For me, (4) is more beneficial than this SPIP because
> > > - We spend the same community effort during voting (including at least
> 3
> > > PMC votes) for (4) and SPIP.
> > > - (4) has the real benefit because users can use it in the production
> > > while SPIP didn't.
> > >
> > > Technically, it could be a little weird if Apache Spark community
> releases
> > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the
> actual
> > > maintenance versions like `4.0.1`, ..., `4.0.2`.
> > >
> > > In short, "Automated Monthly Maintenance Release" might be the
> > > prerequisite for "Monthly Preview Release". What do you think about
> that?
> > > Can we extend your SPIP in this direction?
> > >
> > > Thanks,
> > > Dongjoon.
> > >
> > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
> > >
> > > Hi all,
> > >
> > > I would like to propose a monthly preview for our dev branch, e.g.,
> Spark
> > > 4.1.0 preview1 ... previewN.
> > >
> > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have
> minimized
> > > the manual work so I think it's realistic to propose this.
> > >
> > > Couple of notes:
> > > - The manual steps it requires would be to run GitHub Actions twice
> for RC
> > > and publishing, and summarizing the vote result. There IS a way to even
> > > automate this but it needs more work to comply with ASF policy. I would
> > > like to stick to this minimal manual work for now.
> > > - For now, I would like to volunteer to be responsible for the preview
> > > releases and incrementally improve our release policy guidelines (
> > > https://spark.apache.org/release-process.html) as well once this SPIP
> > > passes.
> > > - The individual release would be, I suspect, about the first week in
> each
> > > month but I would like to avoid setting the explicit date in the SPIP
> so it
> > > makes us less pressured.
> > >
> > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625
> > > SPIP:
> > >
> > >
> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you for the clarification, Hyukjin. Also, thank you for sharing your direction, DB.
I agree with you folks that the AS-IS scope of SPIP is a good start.
+1 for the SPIP because `4.1.0-previewX` itself is actually very helpful already during developing Spark subprojects like "Spark Connect for Swift" and "Spark K8s Operator". :-)
Thank you again.
Dongjoon.
On 2025/07/02 00:31:10 Hyukjin Kwon wrote:
> Hi Dongjoon,
>
> Thanks a lot for your detailed feedback and great questions!
> Let me clarify my current proposal and thoughts:
>
> 1. Regarding Spark 5.0 schedule
> At the moment, I don’t have a concrete Spark 5.0 schedule in mind.
> I included the stable major releases in the Final Success criteria mainly
> to set a practical milestone to complete the automation work and fully
> transition to automated official releases.
> I don't intend to set the next major release timeline in this SPIP.
>
> 2. Lowering the bar for preview releases
> In short, yeah. I expect the bar for preview releases to be lower compared
> to official releases, given that these previews are primarily for early
> testing and feedback.
> That said, if the community raises concerns during the vote and we end up
> with multiple RCs, that’s totally fine. In such cases, we could even skip
> the next month's preview if needed.
> My intention is not to strictly enforce monthly previews but to provide
> regular opportunities for testing, while keeping the process low-pressure
> for the community.
>
> 3. Scope of monthly previews (first minor versions only)
> Yup. This proposal is only for previews of the next minor version from the
> master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until we
> cut the real 4.1.0 release.
> Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on.
> There will be no 4.0.1-preview1 style releases under this proposal.
>
> 4. Official releases (e.g., 4.0.1, 3.5.7)
> For now, this SPIP does not target automating or introducing monthly
> maintenance releases like 4.0.1.
> But yeah, that's my final goal actually. The automated maintenance releases
> are where I want to go next, after proving the automation works reliably
> via previews.
> here are actually some more work to be done to make it actually no manual
> step at all.
>
>
> *TL;DR*: this is a step before automating the official releases (it's not
> tied to the official releases yet to be conservative) + providing users
> with early access to the latest dev Spark build.
>
> On Wed, 2 Jul 2025 at 09:27, DB Tsai <db...@dbtsai.com> wrote:
>
> > Thank you, Hyukjin for driving the SPIP and for your work on the release
> > automation infrastructure — it’s a huge step forward.
> >
> > I’ve been thinking about this topic quite a bit since the Spark 4.0
> > release. While Spark continues to deliver meaningful improvements in every
> > release and enjoys active community contributions, there’s a lingering
> > perception that the project is mature but not evolving quickly. I feel this
> > perception is largely due to the long gap between major versions — it’s
> > been five years between Spark 3.0 and 4.0 — which has understandably caused
> > some frustration among both contributors and users.
> >
> > Now, with release automation and monthly preview builds purposed in this
> > SPIP, we have a real opportunity to change that. As Dongjoon suggested,
> > setting up a regular maintenance release cadence — perhaps bi-monthly
> > instead of monthly — could strike the right balance and make these builds
> > more viable for production environments.
> >
> > If this model proves successful, we could move toward an even faster major
> > release cadence and designate one LTS versions annually, with extended
> > backport support.
> >
> > Benefits for the OSS Community:
> >
> > - Faster time-to-production for new features
> > - Stronger contributor engagement
> > - Quicker community feedback cycles
> > - Easier debugging and testing through smaller, incremental changes
> >
> >
> > Thanks,
> >
> > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> >
> > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org> wrote:
> >
> > Thank you so much for the suggestion and achieving the automated infra,
> > Hyujkin.
> >
> > I have a few questions.
> >
> > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") as
> > "Q8. Final Success" criteria. I'm wondering if you have some schedule in
> > your mind for Spark 5.0 in next 2 years?
> >
> > 2. Are we going to lower the bar for the monthly preview releases?
> > Specifically, I'm wondering if the Preview-RC1 supposed to pass always
> > because it's a preview release? As we know, it's not until now. For
> > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK
> > 4.0.0-preview1 (RC3)".
> >
> > 3. Is SPIP proposing monthly previews for only the FIRST MINOR versions
> > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? There is
> > no `4.0.1-preview1`?
> >
> > 4. Although there was an automated test email for 3.5.7, SPIP is not
> > aiming maintenance version release like 4.0.1 and 3.5.7?
> >
> > Initially, I thought you were going to propose (4) "Automated Monthly
> > Maintenance Release".
> >
> > For me, (4) is more beneficial than this SPIP because
> > - We spend the same community effort during voting (including at least 3
> > PMC votes) for (4) and SPIP.
> > - (4) has the real benefit because users can use it in the production
> > while SPIP didn't.
> >
> > Technically, it could be a little weird if Apache Spark community releases
> > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the actual
> > maintenance versions like `4.0.1`, ..., `4.0.2`.
> >
> > In short, "Automated Monthly Maintenance Release" might be the
> > prerequisite for "Monthly Preview Release". What do you think about that?
> > Can we extend your SPIP in this direction?
> >
> > Thanks,
> > Dongjoon.
> >
> > On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
> >
> > Hi all,
> >
> > I would like to propose a monthly preview for our dev branch, e.g., Spark
> > 4.1.0 preview1 ... previewN.
> >
> > Per https://issues.apache.org/jira/browse/SPARK-52176, we have minimized
> > the manual work so I think it's realistic to propose this.
> >
> > Couple of notes:
> > - The manual steps it requires would be to run GitHub Actions twice for RC
> > and publishing, and summarizing the vote result. There IS a way to even
> > automate this but it needs more work to comply with ASF policy. I would
> > like to stick to this minimal manual work for now.
> > - For now, I would like to volunteer to be responsible for the preview
> > releases and incrementally improve our release policy guidelines (
> > https://spark.apache.org/release-process.html) as well once this SPIP
> > passes.
> > - The individual release would be, I suspect, about the first week in each
> > month but I would like to avoid setting the explicit date in the SPIP so it
> > makes us less pressured.
> >
> > JIRA: https://issues.apache.org/jira/browse/SPARK-52625
> > SPIP:
> >
> > https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Monthly preview release
Posted by Hyukjin Kwon <gu...@apache.org>.
Hi Dongjoon,
Thanks a lot for your detailed feedback and great questions!
Let me clarify my current proposal and thoughts:
1. Regarding Spark 5.0 schedule
At the moment, I don’t have a concrete Spark 5.0 schedule in mind.
I included the stable major releases in the Final Success criteria mainly
to set a practical milestone to complete the automation work and fully
transition to automated official releases.
I don't intend to set the next major release timeline in this SPIP.
2. Lowering the bar for preview releases
In short, yeah. I expect the bar for preview releases to be lower compared
to official releases, given that these previews are primarily for early
testing and feedback.
That said, if the community raises concerns during the vote and we end up
with multiple RCs, that’s totally fine. In such cases, we could even skip
the next month's preview if needed.
My intention is not to strictly enforce monthly previews but to provide
regular opportunities for testing, while keeping the process low-pressure
for the community.
3. Scope of monthly previews (first minor versions only)
Yup. This proposal is only for previews of the next minor version from the
master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until we
cut the real 4.1.0 release.
Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on.
There will be no 4.0.1-preview1 style releases under this proposal.
4. Official releases (e.g., 4.0.1, 3.5.7)
For now, this SPIP does not target automating or introducing monthly
maintenance releases like 4.0.1.
But yeah, that's my final goal actually. The automated maintenance releases
are where I want to go next, after proving the automation works reliably
via previews.
here are actually some more work to be done to make it actually no manual
step at all.
*TL;DR*: this is a step before automating the official releases (it's not
tied to the official releases yet to be conservative) + providing users
with early access to the latest dev Spark build.
On Wed, 2 Jul 2025 at 09:27, DB Tsai <db...@dbtsai.com> wrote:
> Thank you, Hyukjin for driving the SPIP and for your work on the release
> automation infrastructure — it’s a huge step forward.
>
> I’ve been thinking about this topic quite a bit since the Spark 4.0
> release. While Spark continues to deliver meaningful improvements in every
> release and enjoys active community contributions, there’s a lingering
> perception that the project is mature but not evolving quickly. I feel this
> perception is largely due to the long gap between major versions — it’s
> been five years between Spark 3.0 and 4.0 — which has understandably caused
> some frustration among both contributors and users.
>
> Now, with release automation and monthly preview builds purposed in this
> SPIP, we have a real opportunity to change that. As Dongjoon suggested,
> setting up a regular maintenance release cadence — perhaps bi-monthly
> instead of monthly — could strike the right balance and make these builds
> more viable for production environments.
>
> If this model proves successful, we could move toward an even faster major
> release cadence and designate one LTS versions annually, with extended
> backport support.
>
> Benefits for the OSS Community:
>
> - Faster time-to-production for new features
> - Stronger contributor engagement
> - Quicker community feedback cycles
> - Easier debugging and testing through smaller, incremental changes
>
>
> Thanks,
>
> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>
> On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org> wrote:
>
> Thank you so much for the suggestion and achieving the automated infra,
> Hyujkin.
>
> I have a few questions.
>
> 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") as
> "Q8. Final Success" criteria. I'm wondering if you have some schedule in
> your mind for Spark 5.0 in next 2 years?
>
> 2. Are we going to lower the bar for the monthly preview releases?
> Specifically, I'm wondering if the Preview-RC1 supposed to pass always
> because it's a preview release? As we know, it's not until now. For
> example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK
> 4.0.0-preview1 (RC3)".
>
> 3. Is SPIP proposing monthly previews for only the FIRST MINOR versions
> like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? There is
> no `4.0.1-preview1`?
>
> 4. Although there was an automated test email for 3.5.7, SPIP is not
> aiming maintenance version release like 4.0.1 and 3.5.7?
>
> Initially, I thought you were going to propose (4) "Automated Monthly
> Maintenance Release".
>
> For me, (4) is more beneficial than this SPIP because
> - We spend the same community effort during voting (including at least 3
> PMC votes) for (4) and SPIP.
> - (4) has the real benefit because users can use it in the production
> while SPIP didn't.
>
> Technically, it could be a little weird if Apache Spark community releases
> only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the actual
> maintenance versions like `4.0.1`, ..., `4.0.2`.
>
> In short, "Automated Monthly Maintenance Release" might be the
> prerequisite for "Monthly Preview Release". What do you think about that?
> Can we extend your SPIP in this direction?
>
> Thanks,
> Dongjoon.
>
> On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
>
> Hi all,
>
> I would like to propose a monthly preview for our dev branch, e.g., Spark
> 4.1.0 preview1 ... previewN.
>
> Per https://issues.apache.org/jira/browse/SPARK-52176, we have minimized
> the manual work so I think it's realistic to propose this.
>
> Couple of notes:
> - The manual steps it requires would be to run GitHub Actions twice for RC
> and publishing, and summarizing the vote result. There IS a way to even
> automate this but it needs more work to comply with ASF policy. I would
> like to stick to this minimal manual work for now.
> - For now, I would like to volunteer to be responsible for the preview
> releases and incrementally improve our release policy guidelines (
> https://spark.apache.org/release-process.html) as well once this SPIP
> passes.
> - The individual release would be, I suspect, about the first week in each
> month but I would like to avoid setting the explicit date in the SPIP so it
> makes us less pressured.
>
> JIRA: https://issues.apache.org/jira/browse/SPARK-52625
> SPIP:
>
> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
Re: [DISCUSS] SPIP: Monthly preview release
Posted by DB Tsai <db...@dbtsai.com>.
Thank you, Hyukjin for driving the SPIP and for your work on the release automation infrastructure — it’s a huge step forward.
I’ve been thinking about this topic quite a bit since the Spark 4.0 release. While Spark continues to deliver meaningful improvements in every release and enjoys active community contributions, there’s a lingering perception that the project is mature but not evolving quickly. I feel this perception is largely due to the long gap between major versions — it’s been five years between Spark 3.0 and 4.0 — which has understandably caused some frustration among both contributors and users.
Now, with release automation and monthly preview builds purposed in this SPIP, we have a real opportunity to change that. As Dongjoon suggested, setting up a regular maintenance release cadence — perhaps bi-monthly instead of monthly — could strike the right balance and make these builds more viable for production environments.
If this model proves successful, we could move toward an even faster major release cadence and designate one LTS versions annually, with extended backport support.
Benefits for the OSS Community:
Faster time-to-production for new features
Stronger contributor engagement
Quicker community feedback cycles
Easier debugging and testing through smaller, incremental changes
Thanks,
DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <do...@apache.org> wrote:
>
> Thank you so much for the suggestion and achieving the automated infra, Hyujkin.
>
> I have a few questions.
>
> 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") as "Q8. Final Success" criteria. I'm wondering if you have some schedule in your mind for Spark 5.0 in next 2 years?
>
> 2. Are we going to lower the bar for the monthly preview releases? Specifically, I'm wondering if the Preview-RC1 supposed to pass always because it's a preview release? As we know, it's not until now. For example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK 4.0.0-preview1 (RC3)".
>
> 3. Is SPIP proposing monthly previews for only the FIRST MINOR versions like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? There is no `4.0.1-preview1`?
>
> 4. Although there was an automated test email for 3.5.7, SPIP is not aiming maintenance version release like 4.0.1 and 3.5.7?
>
> Initially, I thought you were going to propose (4) "Automated Monthly Maintenance Release".
>
> For me, (4) is more beneficial than this SPIP because
> - We spend the same community effort during voting (including at least 3 PMC votes) for (4) and SPIP.
> - (4) has the real benefit because users can use it in the production while SPIP didn't.
>
> Technically, it could be a little weird if Apache Spark community releases only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the actual maintenance versions like `4.0.1`, ..., `4.0.2`.
>
> In short, "Automated Monthly Maintenance Release" might be the prerequisite for "Monthly Preview Release". What do you think about that? Can we extend your SPIP in this direction?
>
> Thanks,
> Dongjoon.
>
> On 2025/06/30 23:34:54 Hyukjin Kwon wrote:
>> Hi all,
>>
>> I would like to propose a monthly preview for our dev branch, e.g., Spark
>> 4.1.0 preview1 ... previewN.
>>
>> Per https://issues.apache.org/jira/browse/SPARK-52176, we have minimized
>> the manual work so I think it's realistic to propose this.
>>
>> Couple of notes:
>> - The manual steps it requires would be to run GitHub Actions twice for RC
>> and publishing, and summarizing the vote result. There IS a way to even
>> automate this but it needs more work to comply with ASF policy. I would
>> like to stick to this minimal manual work for now.
>> - For now, I would like to volunteer to be responsible for the preview
>> releases and incrementally improve our release policy guidelines (
>> https://spark.apache.org/release-process.html) as well once this SPIP
>> passes.
>> - The individual release would be, I suspect, about the first week in each
>> month but I would like to avoid setting the explicit date in the SPIP so it
>> makes us less pressured.
>>
>> JIRA: https://issues.apache.org/jira/browse/SPARK-52625
>> SPIP:
>> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>