You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dongjoon Hyun <do...@apache.org> on 2023/06/01 01:02:41 UTC
Apache Spark 4.0 Timeframe?
Hi, All.
I'd like to propose to start to prepare Apache Spark 4.0 after creating
branch-3.5 on July 16th.
- https://spark.apache.org/versioning-policy.html
Historically, the Apache Spark release dates have the following timeframes
and we already have Spark 3.5 plan which will be maintained up to 2026.
Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
Spark 4: 2024.06 (4.0.0, NEW)
As we discussed in the previous email thread, `Apache Spark 3.5.0
Expectations`, we cannot deliver some features without Apache Spark 4.
- "I wonder if it’s safer to do it in Spark 4 (which I believe will be
discussed soon)."
- "I would make it the default at 4.0, myself."
Although there exist more other features, let's focus on Scala language
support history.
Spark 2.0: SPARK-6363 Make Scala 2.11 the default Scala version (2016.07)
Spark 3.0: SPARK-25956 Make Scala 2.12 as default Scala version in Spark
3.0 (2020.06)
In addition, the Scala community released Scala 3.3.0 LTS yesterday.
- https://scala-lang.org/blog/2023/05/30/scala-3.3.0-released.html
If we decide to start, I believe we can support Scala 2.13 or Scala 3.3
next year with Apache Spark 4 while supporting Spark 3.4 and 3.5 for Scala
2.12 users.
WDYT?
Thanks,
Dongjoon.
Re: Apache Spark 4.0 Timeframe?
Posted by Dongjoon Hyun <do...@apache.org>.
It's great to hear from you that you are interested in the discussion of
Apache Spark 4.0 scope and timeframe. :)
This is the initial thread which includes upgrading the default Scala
version to 2.13 or 3.3 is the one of the reasons.
As we know, Apache Spark 4.0 is not limited to this. For example, dropping
Java 8 is also discussed in "JDK version support policy?" thread separately.
In principle, this thread aims to avoid blocking the Apache Spark community
by saying 'No for all; (1) No until Apache Spark 4.0 and (2) No 4.0 because
we don't have much".
So, could you share your opinion for Apache Spark 4.0 here?
Best,
Dongjoon.
On Wed, May 31, 2023 at 6:02 PM Dongjoon Hyun <do...@apache.org> wrote:
> Hi, All.
>
> I'd like to propose to start to prepare Apache Spark 4.0 after creating
> branch-3.5 on July 16th.
>
> - https://spark.apache.org/versioning-policy.html
>
> Historically, the Apache Spark release dates have the following timeframes
> and we already have Spark 3.5 plan which will be maintained up to 2026.
>
> Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
> Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
> Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
> Spark 4: 2024.06 (4.0.0, NEW)
>
> As we discussed in the previous email thread, `Apache Spark 3.5.0
> Expectations`, we cannot deliver some features without Apache Spark 4.
>
> - "I wonder if it’s safer to do it in Spark 4 (which I believe will be
> discussed soon)."
> - "I would make it the default at 4.0, myself."
>
> Although there exist more other features, let's focus on Scala language
> support history.
>
> Spark 2.0: SPARK-6363 Make Scala 2.11 the default Scala version (2016.07)
> Spark 3.0: SPARK-25956 Make Scala 2.12 as default Scala version in Spark
> 3.0 (2020.06)
>
> In addition, the Scala community released Scala 3.3.0 LTS yesterday.
>
> - https://scala-lang.org/blog/2023/05/30/scala-3.3.0-released.html
>
> If we decide to start, I believe we can support Scala 2.13 or Scala 3.3
> next year with Apache Spark 4 while supporting Spark 3.4 and 3.5 for Scala
> 2.12 users.
>
> WDYT?
>
> Thanks,
> Dongjoon.
>
Re: Apache Spark 4.0 Timeframe?
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you, Yangjie.
Yes, instead of switching, it will depend on the Scala community's promised
compatibility completely.
However, many other difficulties still arise from the 3rd-party libraries
like Ammonite. Scala 3 might be better in long term perspective.
I'll send another email about the community Scala issues.
Dongjoon.
On Fri, Jun 2, 2023 at 2:48 AM yangjie01 <ya...@baidu.com> wrote:
> +1，Agree to start to prepare Apache Spark 4.0 after creating branch-3.5
> on July 16th.
>
>
>
> As I am not yet familiar with Scala 3, I am unable to make good
> suggestions for choosing the Scala version.
>
>
>
> But I want to know if Spark 4.0 chooses to use the Scala 2.13.x, is it
> impossible to switch Scala 3.x as the default version during the lifecycle
> of Spark 4.x?
>
>
>
> Thanks
>
> Yang Jie
>
>
>
> *发件人**: *Dongjoon Hyun <do...@apache.org>
> *日期**: *2023年6月1日 星期四 09:03
> *收件人**: *dev <de...@spark.apache.org>
> *主题**: *Apache Spark 4.0 Timeframe?
>
>
>
> Hi, All.
>
> I'd like to propose to start to prepare Apache Spark 4.0 after creating
> branch-3.5 on July 16th.
>
> - https://spark.apache.org/versioning-policy.html
> <https://mailshield.baidu.com/check?q=vctdW221Qpw1RkBH%2baEmJJYTNfyfKdQth%2bXybE4CmiqqiIKDFEKSh0Cm4i6jy9T6ROH2vg%3d%3d>
>
> Historically, the Apache Spark release dates have the following timeframes
> and we already have Spark 3.5 plan which will be maintained up to 2026.
>
> Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
> Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
> Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
> Spark 4: 2024.06 (4.0.0, NEW)
>
> As we discussed in the previous email thread, `Apache Spark 3.5.0
> Expectations`, we cannot deliver some features without Apache Spark 4.
>
> - "I wonder if it’s safer to do it in Spark 4 (which I believe will be
> discussed soon)."
> - "I would make it the default at 4.0, myself."
>
> Although there exist more other features, let's focus on Scala language
> support history.
>
> Spark 2.0: SPARK-6363 Make Scala 2.11 the default Scala version (2016.07)
> Spark 3.0: SPARK-25956 Make Scala 2.12 as default Scala version in Spark
> 3.0 (2020.06)
>
> In addition, the Scala community released Scala 3.3.0 LTS yesterday.
>
> - https://scala-lang.org/blog/2023/05/30/scala-3.3.0-released.html
> <https://mailshield.baidu.com/check?q=I4UJVy3PdUFSU6r495P905XswiaNsj37cgObnPREo6HrPJsFLSIOaS5E9Axx%2ftIaBSbs0VK9C4MpgDtorx2cAVibNDQ%3d>
>
> If we decide to start, I believe we can support Scala 2.13 or Scala 3.3
> next year with Apache Spark 4 while supporting Spark 3.4 and 3.5 for Scala
> 2.12 users.
>
> WDYT?
>
> Thanks,
> Dongjoon.
>
Re: Apache Spark 4.0 Timeframe?
Posted by yangjie01 <ya...@baidu.com>.
+1，Agree to start to prepare Apache Spark 4.0 after creating branch-3.5 on July 16th.
As I am not yet familiar with Scala 3, I am unable to make good suggestions for choosing the Scala version.
But I want to know if Spark 4.0 chooses to use the Scala 2.13.x, is it impossible to switch Scala 3.x as the default version during the lifecycle of Spark 4.x?
Thanks
Yang Jie
发件人: Dongjoon Hyun <do...@apache.org>
日期: 2023年6月1日 星期四 09:03
收件人: dev <de...@spark.apache.org>
主题: Apache Spark 4.0 Timeframe?
Hi, All.
I'd like to propose to start to prepare Apache Spark 4.0 after creating branch-3.5 on July 16th.
- https://spark.apache.org/versioning-policy.html<https://mailshield.baidu.com/check?q=vctdW221Qpw1RkBH%2baEmJJYTNfyfKdQth%2bXybE4CmiqqiIKDFEKSh0Cm4i6jy9T6ROH2vg%3d%3d>
Historically, the Apache Spark release dates have the following timeframes and we already have Spark 3.5 plan which will be maintained up to 2026.
Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
Spark 4: 2024.06 (4.0.0, NEW)
As we discussed in the previous email thread, `Apache Spark 3.5.0 Expectations`, we cannot deliver some features without Apache Spark 4.
- "I wonder if it’s safer to do it in Spark 4 (which I believe will be discussed soon)."
- "I would make it the default at 4.0, myself."
Although there exist more other features, let's focus on Scala language support history.
Spark 2.0: SPARK-6363 Make Scala 2.11 the default Scala version (2016.07)
Spark 3.0: SPARK-25956 Make Scala 2.12 as default Scala version in Spark 3.0 (2020.06)
In addition, the Scala community released Scala 3.3.0 LTS yesterday.
- https://scala-lang.org/blog/2023/05/30/scala-3.3.0-released.html<https://mailshield.baidu.com/check?q=I4UJVy3PdUFSU6r495P905XswiaNsj37cgObnPREo6HrPJsFLSIOaS5E9Axx%2ftIaBSbs0VK9C4MpgDtorx2cAVibNDQ%3d>
If we decide to start, I believe we can support Scala 2.13 or Scala 3.3 next year with Apache Spark 4 while supporting Spark 3.4 and 3.5 for Scala 2.12 users.
WDYT?
Thanks,
Dongjoon.