You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yuming Wang <wg...@gmail.com> on 2022/09/12 13:37:27 UTC
Time for Spark 3.3.1 release?
Hi, All.
Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches including 7
correctness patches arrived at branch-3.3.
Shall we make a new release, Apache Spark 3.3.1, as the second release at
branch-3.3? I'd like to volunteer as the release manager for Apache Spark 3.
3.1.
All changes:
https://github.com/apache/spark/compare/v3.3.0...branch-3.3
Correctness issues:
SPARK-40149: Propagate metadata columns through Project
SPARK-40002: Don't push down limit through window using ntile
SPARK-39976: ArrayIntersect should handle null in left expression correctly
SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness
issue in the case of overlapping partition and data columns
SPARK-39061: Set nullable correctly for Inline output attributes
SPARK-39887: RemoveRedundantAliases should keep aliases that make the
output of projection nodes unique
SPARK-38614: Don't push down limit through window that's using percent_rank
Re: Time for Spark 3.3.1 release?
Posted by Dongjoon Hyun <do...@gmail.com>.
Although it's irrelevant to Apache Spark 3.3.1 release discussion because
3.3.1 is a maintenance release to 3.3.0, you may want to lead it for Apache
Spark 3.4 in a separate thread. For your info, Apache Spark 3.3.1 RC1 does
not include Hadoop 3.3.4 either.
Previously, since we don't want to introduce any risks (or regressions) due
to new Hadoop2 changes, we started to distribute
Hadoop3-distribution additionally and have been enhancing it. As of today,
we recommend to use Hadoop3 distributions on all environments, or recommend
to build custom distributions based on the user environments (if Hadoop3
distribution is not applicable).
Apache Spark community has been highly interested in the blockers where the
users cannot use the official Hadoop3 distribution in user environments.
Please let us know if there exist issues.
Dongjoon.
On Wed, Sep 14, 2022 at 11:42 AM Bjørn Jørgensen <bj...@gmail.com>
wrote:
> At least we should upgrade hadoop to the latest version
> https://hadoop.apache.org/release/2.10.2.html
>
> Are there some spesial reasons why we have a hadoop version that is 7
> years old?
>
> ons. 14. sep. 2022, 20:25 skrev Dongjoon Hyun <do...@gmail.com>:
>
>> Ya, +1 for Sean's comment.
>>
>> In addition, all Apache Spark's Maven artifacts are depending on Hadoop
>> 3.3.x already.
>>
>>
>> https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.3.0
>>
>> https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.3.0
>>
>> Apache Spark has been moving away from Hadoop 2 due to many many reasons.
>>
>> Dongjoon.
>>
>>
>> On Wed, Sep 14, 2022 at 10:54 AM Sean Owen <sr...@gmail.com> wrote:
>>
>>> Yeah we're not going to make convenience binaries for all possible
>>> combinations. It's a pretty good assumption that anyone moving to later
>>> Scala versions is also off old Hadoop versions.
>>> You can of course build the combo you like.
>>>
>>> On Wed, Sep 14, 2022 at 11:26 AM Denis Bolshakov <
>>> bolshakov.denis@gmail.com> wrote:
>>>
>>>> Unfortunately it's for hadoop 3 only.
>>>>
>>>> ср, 14 сент. 2022 г., 19:04 Dongjoon Hyun <do...@gmail.com>:
>>>>
>>>>> Hi, Denis.
>>>>>
>>>>> Apache Spark community already provides both Scala 2.12 and 2.13
>>>>> pre-built distributions.
>>>>> Please check the distribution site and Apache Spark download page.
>>>>>
>>>>> https://dlcdn.apache.org/spark/spark-3.3.0/
>>>>>
>>>>> spark-3.3.0-bin-hadoop3-scala2.13.tgz
>>>>> spark-3.3.0-bin-hadoop3.tgz
>>>>>
>>>>> [image: Screenshot 2022-09-14 at 9.03.27 AM.png]
>>>>>
>>>>> Dongjoon.
>>>>>
>>>>> On Wed, Sep 14, 2022 at 12:31 AM Denis Bolshakov <
>>>>> bolshakov.denis@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> It would be great if it's possible to provide a spark distro for both
>>>>>> scala 2.12 and scala 2.13.
>>>>>>
>>>>>> It will encourage spark users to switch to scala 2.13.
>>>>>>
>>>>>> I know that spark jar artifacts available for both scala versions,
>>>>>> but it does not make sense to migrate to scala 2.13 while there is no spark
>>>>>> distro for this version.
>>>>>>
>>>>>> Kind regards,
>>>>>> Denis
>>>>>>
>>>>>> On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wg...@gmail.com> wrote:
>>>>>>
>>>>>>> Thank you all.
>>>>>>>
>>>>>>> I will be preparing 3.3.1 RC1 soon.
>>>>>>>
>>>>>>> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks Yuming ~
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>>>>>>>>> *日期**: *2022年9月13日 星期二 08:19
>>>>>>>>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>>>>>>>>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>>>>>>>>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>>>>>>>>> dev@spark.apache.org>
>>>>>>>>> *主题**: *Re: Time for Spark 3.3.1 release?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> +1.
>>>>>>>>>
>>>>>>>>> Thank you, Yuming!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> Thanks Yuming!
>>>>>>>>>
>>>>>>>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <
>>>>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > +1
>>>>>>>>> >
>>>>>>>>> > Thanks,
>>>>>>>>> > Dongjoon.
>>>>>>>>> >
>>>>>>>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Hi, All.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>>>>>>>>> including 7 correctness patches arrived at branch-3.3.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second
>>>>>>>>> release at branch-3.3? I'd like to volunteer as the release manager for
>>>>>>>>> Apache Spark 3.3.1.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> All changes:
>>>>>>>>> >>
>>>>>>>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>>>>>>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> Correctness issues:
>>>>>>>>> >>
>>>>>>>>> >> SPARK-40149: Propagate metadata columns through Project
>>>>>>>>> >>
>>>>>>>>> >> SPARK-40002: Don't push down limit through window using ntile
>>>>>>>>> >>
>>>>>>>>> >> SPARK-39976: ArrayIntersect should handle null in left
>>>>>>>>> expression correctly
>>>>>>>>> >>
>>>>>>>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a
>>>>>>>>> correctness issue in the case of overlapping partition and data columns
>>>>>>>>> >>
>>>>>>>>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>>>>>>>>> >>
>>>>>>>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that
>>>>>>>>> make the output of projection nodes unique
>>>>>>>>> >>
>>>>>>>>> >> SPARK-38614: Don't push down limit through window that's using
>>>>>>>>> percent_rank
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> John Zhuge
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> //with Best Regards
>>>>>> --Denis Bolshakov
>>>>>> e-mail: bolshakov.denis@gmail.com
>>>>>>
>>>>>
Re: Time for Spark 3.3.1 release?
Posted by Bjørn Jørgensen <bj...@gmail.com>.
At least we should upgrade hadoop to the latest version
https://hadoop.apache.org/release/2.10.2.html
Are there some spesial reasons why we have a hadoop version that is 7
years old?
ons. 14. sep. 2022, 20:25 skrev Dongjoon Hyun <do...@gmail.com>:
> Ya, +1 for Sean's comment.
>
> In addition, all Apache Spark's Maven artifacts are depending on Hadoop
> 3.3.x already.
>
>
> https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.3.0
>
> https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.3.0
>
> Apache Spark has been moving away from Hadoop 2 due to many many reasons.
>
> Dongjoon.
>
>
> On Wed, Sep 14, 2022 at 10:54 AM Sean Owen <sr...@gmail.com> wrote:
>
>> Yeah we're not going to make convenience binaries for all possible
>> combinations. It's a pretty good assumption that anyone moving to later
>> Scala versions is also off old Hadoop versions.
>> You can of course build the combo you like.
>>
>> On Wed, Sep 14, 2022 at 11:26 AM Denis Bolshakov <
>> bolshakov.denis@gmail.com> wrote:
>>
>>> Unfortunately it's for hadoop 3 only.
>>>
>>> ср, 14 сент. 2022 г., 19:04 Dongjoon Hyun <do...@gmail.com>:
>>>
>>>> Hi, Denis.
>>>>
>>>> Apache Spark community already provides both Scala 2.12 and 2.13
>>>> pre-built distributions.
>>>> Please check the distribution site and Apache Spark download page.
>>>>
>>>> https://dlcdn.apache.org/spark/spark-3.3.0/
>>>>
>>>> spark-3.3.0-bin-hadoop3-scala2.13.tgz
>>>> spark-3.3.0-bin-hadoop3.tgz
>>>>
>>>> [image: Screenshot 2022-09-14 at 9.03.27 AM.png]
>>>>
>>>> Dongjoon.
>>>>
>>>> On Wed, Sep 14, 2022 at 12:31 AM Denis Bolshakov <
>>>> bolshakov.denis@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> It would be great if it's possible to provide a spark distro for both
>>>>> scala 2.12 and scala 2.13.
>>>>>
>>>>> It will encourage spark users to switch to scala 2.13.
>>>>>
>>>>> I know that spark jar artifacts available for both scala versions, but
>>>>> it does not make sense to migrate to scala 2.13 while there is no spark
>>>>> distro for this version.
>>>>>
>>>>> Kind regards,
>>>>> Denis
>>>>>
>>>>> On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wg...@gmail.com> wrote:
>>>>>
>>>>>> Thank you all.
>>>>>>
>>>>>> I will be preparing 3.3.1 RC1 soon.
>>>>>>
>>>>>> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks Yuming ~
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>>>>>>>> *日期**: *2022年9月13日 星期二 08:19
>>>>>>>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>>>>>>>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>>>>>>>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>>>>>>>> dev@spark.apache.org>
>>>>>>>> *主题**: *Re: Time for Spark 3.3.1 release?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> +1.
>>>>>>>>
>>>>>>>> Thank you, Yuming!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> Thanks Yuming!
>>>>>>>>
>>>>>>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <
>>>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > +1
>>>>>>>> >
>>>>>>>> > Thanks,
>>>>>>>> > Dongjoon.
>>>>>>>> >
>>>>>>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> Hi, All.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>>>>>>>> including 7 correctness patches arrived at branch-3.3.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second
>>>>>>>> release at branch-3.3? I'd like to volunteer as the release manager for
>>>>>>>> Apache Spark 3.3.1.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> All changes:
>>>>>>>> >>
>>>>>>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>>>>>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> Correctness issues:
>>>>>>>> >>
>>>>>>>> >> SPARK-40149: Propagate metadata columns through Project
>>>>>>>> >>
>>>>>>>> >> SPARK-40002: Don't push down limit through window using ntile
>>>>>>>> >>
>>>>>>>> >> SPARK-39976: ArrayIntersect should handle null in left
>>>>>>>> expression correctly
>>>>>>>> >>
>>>>>>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a
>>>>>>>> correctness issue in the case of overlapping partition and data columns
>>>>>>>> >>
>>>>>>>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>>>>>>>> >>
>>>>>>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that
>>>>>>>> make the output of projection nodes unique
>>>>>>>> >>
>>>>>>>> >> SPARK-38614: Don't push down limit through window that's using
>>>>>>>> percent_rank
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>> --
>>>>>>> John Zhuge
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> //with Best Regards
>>>>> --Denis Bolshakov
>>>>> e-mail: bolshakov.denis@gmail.com
>>>>>
>>>>
Re: Time for Spark 3.3.1 release?
Posted by Dongjoon Hyun <do...@gmail.com>.
Ya, +1 for Sean's comment.
In addition, all Apache Spark's Maven artifacts are depending on Hadoop
3.3.x already.
https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.3.0
https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.3.0
Apache Spark has been moving away from Hadoop 2 due to many many reasons.
Dongjoon.
On Wed, Sep 14, 2022 at 10:54 AM Sean Owen <sr...@gmail.com> wrote:
> Yeah we're not going to make convenience binaries for all possible
> combinations. It's a pretty good assumption that anyone moving to later
> Scala versions is also off old Hadoop versions.
> You can of course build the combo you like.
>
> On Wed, Sep 14, 2022 at 11:26 AM Denis Bolshakov <
> bolshakov.denis@gmail.com> wrote:
>
>> Unfortunately it's for hadoop 3 only.
>>
>> ср, 14 сент. 2022 г., 19:04 Dongjoon Hyun <do...@gmail.com>:
>>
>>> Hi, Denis.
>>>
>>> Apache Spark community already provides both Scala 2.12 and 2.13
>>> pre-built distributions.
>>> Please check the distribution site and Apache Spark download page.
>>>
>>> https://dlcdn.apache.org/spark/spark-3.3.0/
>>>
>>> spark-3.3.0-bin-hadoop3-scala2.13.tgz
>>> spark-3.3.0-bin-hadoop3.tgz
>>>
>>> [image: Screenshot 2022-09-14 at 9.03.27 AM.png]
>>>
>>> Dongjoon.
>>>
>>> On Wed, Sep 14, 2022 at 12:31 AM Denis Bolshakov <
>>> bolshakov.denis@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> It would be great if it's possible to provide a spark distro for both
>>>> scala 2.12 and scala 2.13.
>>>>
>>>> It will encourage spark users to switch to scala 2.13.
>>>>
>>>> I know that spark jar artifacts available for both scala versions, but
>>>> it does not make sense to migrate to scala 2.13 while there is no spark
>>>> distro for this version.
>>>>
>>>> Kind regards,
>>>> Denis
>>>>
>>>> On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wg...@gmail.com> wrote:
>>>>
>>>>> Thank you all.
>>>>>
>>>>> I will be preparing 3.3.1 RC1 soon.
>>>>>
>>>>> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks Yuming ~
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>>>>>>> *日期**: *2022年9月13日 星期二 08:19
>>>>>>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>>>>>>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>>>>>>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>>>>>>> dev@spark.apache.org>
>>>>>>> *主题**: *Re: Time for Spark 3.3.1 release?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> +1.
>>>>>>>
>>>>>>> Thank you, Yuming!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> Thanks Yuming!
>>>>>>>
>>>>>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <
>>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>>> >
>>>>>>> > +1
>>>>>>> >
>>>>>>> > Thanks,
>>>>>>> > Dongjoon.
>>>>>>> >
>>>>>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Hi, All.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>>>>>>> including 7 correctness patches arrived at branch-3.3.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second
>>>>>>> release at branch-3.3? I'd like to volunteer as the release manager for
>>>>>>> Apache Spark 3.3.1.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> All changes:
>>>>>>> >>
>>>>>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>>>>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Correctness issues:
>>>>>>> >>
>>>>>>> >> SPARK-40149: Propagate metadata columns through Project
>>>>>>> >>
>>>>>>> >> SPARK-40002: Don't push down limit through window using ntile
>>>>>>> >>
>>>>>>> >> SPARK-39976: ArrayIntersect should handle null in left expression
>>>>>>> correctly
>>>>>>> >>
>>>>>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a
>>>>>>> correctness issue in the case of overlapping partition and data columns
>>>>>>> >>
>>>>>>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>>>>>>> >>
>>>>>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make
>>>>>>> the output of projection nodes unique
>>>>>>> >>
>>>>>>> >> SPARK-38614: Don't push down limit through window that's using
>>>>>>> percent_rank
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>> --
>>>>>> John Zhuge
>>>>>>
>>>>>
>>>>
>>>> --
>>>> //with Best Regards
>>>> --Denis Bolshakov
>>>> e-mail: bolshakov.denis@gmail.com
>>>>
>>>
Re: Time for Spark 3.3.1 release?
Posted by Sean Owen <sr...@gmail.com>.
Yeah we're not going to make convenience binaries for all possible
combinations. It's a pretty good assumption that anyone moving to later
Scala versions is also off old Hadoop versions.
You can of course build the combo you like.
On Wed, Sep 14, 2022 at 11:26 AM Denis Bolshakov <bo...@gmail.com>
wrote:
> Unfortunately it's for hadoop 3 only.
>
> ср, 14 сент. 2022 г., 19:04 Dongjoon Hyun <do...@gmail.com>:
>
>> Hi, Denis.
>>
>> Apache Spark community already provides both Scala 2.12 and 2.13
>> pre-built distributions.
>> Please check the distribution site and Apache Spark download page.
>>
>> https://dlcdn.apache.org/spark/spark-3.3.0/
>>
>> spark-3.3.0-bin-hadoop3-scala2.13.tgz
>> spark-3.3.0-bin-hadoop3.tgz
>>
>> [image: Screenshot 2022-09-14 at 9.03.27 AM.png]
>>
>> Dongjoon.
>>
>> On Wed, Sep 14, 2022 at 12:31 AM Denis Bolshakov <
>> bolshakov.denis@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> It would be great if it's possible to provide a spark distro for both
>>> scala 2.12 and scala 2.13.
>>>
>>> It will encourage spark users to switch to scala 2.13.
>>>
>>> I know that spark jar artifacts available for both scala versions, but
>>> it does not make sense to migrate to scala 2.13 while there is no spark
>>> distro for this version.
>>>
>>> Kind regards,
>>> Denis
>>>
>>> On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wg...@gmail.com> wrote:
>>>
>>>> Thank you all.
>>>>
>>>> I will be preparing 3.3.1 RC1 soon.
>>>>
>>>> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks Yuming ~
>>>>>>
>>>>>>
>>>>>>
>>>>>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>>>>>> *日期**: *2022年9月13日 星期二 08:19
>>>>>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>>>>>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>>>>>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>>>>>> dev@spark.apache.org>
>>>>>> *主题**: *Re: Time for Spark 3.3.1 release?
>>>>>>
>>>>>>
>>>>>>
>>>>>> +1
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> +1.
>>>>>>
>>>>>> Thank you, Yuming!
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> Thanks Yuming!
>>>>>>
>>>>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <
>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>> >
>>>>>> > +1
>>>>>> >
>>>>>> > Thanks,
>>>>>> > Dongjoon.
>>>>>> >
>>>>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Hi, All.
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>>>>>> including 7 correctness patches arrived at branch-3.3.
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second
>>>>>> release at branch-3.3? I'd like to volunteer as the release manager for
>>>>>> Apache Spark 3.3.1.
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> All changes:
>>>>>> >>
>>>>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>>>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> Correctness issues:
>>>>>> >>
>>>>>> >> SPARK-40149: Propagate metadata columns through Project
>>>>>> >>
>>>>>> >> SPARK-40002: Don't push down limit through window using ntile
>>>>>> >>
>>>>>> >> SPARK-39976: ArrayIntersect should handle null in left expression
>>>>>> correctly
>>>>>> >>
>>>>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a
>>>>>> correctness issue in the case of overlapping partition and data columns
>>>>>> >>
>>>>>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>>>>>> >>
>>>>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make
>>>>>> the output of projection nodes unique
>>>>>> >>
>>>>>> >> SPARK-38614: Don't push down limit through window that's using
>>>>>> percent_rank
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>> --
>>>>> John Zhuge
>>>>>
>>>>
>>>
>>> --
>>> //with Best Regards
>>> --Denis Bolshakov
>>> e-mail: bolshakov.denis@gmail.com
>>>
>>
Re: Time for Spark 3.3.1 release?
Posted by Denis Bolshakov <bo...@gmail.com>.
Unfortunately it's for hadoop 3 only.
ср, 14 сент. 2022 г., 19:04 Dongjoon Hyun <do...@gmail.com>:
> Hi, Denis.
>
> Apache Spark community already provides both Scala 2.12 and 2.13 pre-built
> distributions.
> Please check the distribution site and Apache Spark download page.
>
> https://dlcdn.apache.org/spark/spark-3.3.0/
>
> spark-3.3.0-bin-hadoop3-scala2.13.tgz
> spark-3.3.0-bin-hadoop3.tgz
>
> [image: Screenshot 2022-09-14 at 9.03.27 AM.png]
>
> Dongjoon.
>
> On Wed, Sep 14, 2022 at 12:31 AM Denis Bolshakov <
> bolshakov.denis@gmail.com> wrote:
>
>> Hello,
>>
>> It would be great if it's possible to provide a spark distro for both
>> scala 2.12 and scala 2.13.
>>
>> It will encourage spark users to switch to scala 2.13.
>>
>> I know that spark jar artifacts available for both scala versions, but it
>> does not make sense to migrate to scala 2.13 while there is no spark distro
>> for this version.
>>
>> Kind regards,
>> Denis
>>
>> On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wg...@gmail.com> wrote:
>>
>>> Thank you all.
>>>
>>> I will be preparing 3.3.1 RC1 soon.
>>>
>>> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> +1
>>>>
>>>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>>
>>>>> Thanks Yuming ~
>>>>>
>>>>>
>>>>>
>>>>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>>>>> *日期**: *2022年9月13日 星期二 08:19
>>>>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>>>>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>>>>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>>>>> dev@spark.apache.org>
>>>>> *主题**: *Re: Time for Spark 3.3.1 release?
>>>>>
>>>>>
>>>>>
>>>>> +1
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> +1.
>>>>>
>>>>> Thank you, Yuming!
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>>
>>>>> +1
>>>>>
>>>>> Thanks Yuming!
>>>>>
>>>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <
>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>> >
>>>>> > +1
>>>>> >
>>>>> > Thanks,
>>>>> > Dongjoon.
>>>>> >
>>>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Hi, All.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>>>>> including 7 correctness patches arrived at branch-3.3.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second
>>>>> release at branch-3.3? I'd like to volunteer as the release manager for
>>>>> Apache Spark 3.3.1.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> All changes:
>>>>> >>
>>>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> Correctness issues:
>>>>> >>
>>>>> >> SPARK-40149: Propagate metadata columns through Project
>>>>> >>
>>>>> >> SPARK-40002: Don't push down limit through window using ntile
>>>>> >>
>>>>> >> SPARK-39976: ArrayIntersect should handle null in left expression
>>>>> correctly
>>>>> >>
>>>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a
>>>>> correctness issue in the case of overlapping partition and data columns
>>>>> >>
>>>>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>>>>> >>
>>>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make
>>>>> the output of projection nodes unique
>>>>> >>
>>>>> >> SPARK-38614: Don't push down limit through window that's using
>>>>> percent_rank
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>> --
>>>> John Zhuge
>>>>
>>>
>>
>> --
>> //with Best Regards
>> --Denis Bolshakov
>> e-mail: bolshakov.denis@gmail.com
>>
>
Re: Time for Spark 3.3.1 release?
Posted by Dongjoon Hyun <do...@gmail.com>.
Hi, Denis.
Apache Spark community already provides both Scala 2.12 and 2.13 pre-built
distributions.
Please check the distribution site and Apache Spark download page.
https://dlcdn.apache.org/spark/spark-3.3.0/
spark-3.3.0-bin-hadoop3-scala2.13.tgz
spark-3.3.0-bin-hadoop3.tgz
[image: Screenshot 2022-09-14 at 9.03.27 AM.png]
Dongjoon.
On Wed, Sep 14, 2022 at 12:31 AM Denis Bolshakov <bo...@gmail.com>
wrote:
> Hello,
>
> It would be great if it's possible to provide a spark distro for both
> scala 2.12 and scala 2.13.
>
> It will encourage spark users to switch to scala 2.13.
>
> I know that spark jar artifacts available for both scala versions, but it
> does not make sense to migrate to scala 2.13 while there is no spark distro
> for this version.
>
> Kind regards,
> Denis
>
> On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wg...@gmail.com> wrote:
>
>> Thank you all.
>>
>> I will be preparing 3.3.1 RC1 soon.
>>
>> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org> wrote:
>>
>>> +1
>>>
>>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>>
>>>> Thanks Yuming ~
>>>>
>>>>
>>>>
>>>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>>>> *日期**: *2022年9月13日 星期二 08:19
>>>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>>>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>>>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>>>> dev@spark.apache.org>
>>>> *主题**: *Re: Time for Spark 3.3.1 release?
>>>>
>>>>
>>>>
>>>> +1
>>>>
>>>>
>>>>
>>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com> wrote:
>>>>
>>>> +1.
>>>>
>>>> Thank you, Yuming!
>>>>
>>>>
>>>>
>>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>
>>>> +1
>>>>
>>>> Thanks Yuming!
>>>>
>>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com>
>>>> wrote:
>>>> >
>>>> > +1
>>>> >
>>>> > Thanks,
>>>> > Dongjoon.
>>>> >
>>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
>>>> >>
>>>> >> Hi, All.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>>>> including 7 correctness patches arrived at branch-3.3.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second
>>>> release at branch-3.3? I'd like to volunteer as the release manager for
>>>> Apache Spark 3.3.1.
>>>> >>
>>>> >>
>>>> >>
>>>> >> All changes:
>>>> >>
>>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>>> >>
>>>> >>
>>>> >>
>>>> >> Correctness issues:
>>>> >>
>>>> >> SPARK-40149: Propagate metadata columns through Project
>>>> >>
>>>> >> SPARK-40002: Don't push down limit through window using ntile
>>>> >>
>>>> >> SPARK-39976: ArrayIntersect should handle null in left expression
>>>> correctly
>>>> >>
>>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a
>>>> correctness issue in the case of overlapping partition and data columns
>>>> >>
>>>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>>>> >>
>>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make
>>>> the output of projection nodes unique
>>>> >>
>>>> >> SPARK-38614: Don't push down limit through window that's using
>>>> percent_rank
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>> --
>>> John Zhuge
>>>
>>
>
> --
> //with Best Regards
> --Denis Bolshakov
> e-mail: bolshakov.denis@gmail.com
>
Re: Time for Spark 3.3.1 release?
Posted by Denis Bolshakov <bo...@gmail.com>.
Hello,
It would be great if it's possible to provide a spark distro for both scala
2.12 and scala 2.13.
It will encourage spark users to switch to scala 2.13.
I know that spark jar artifacts available for both scala versions, but it
does not make sense to migrate to scala 2.13 while there is no spark distro
for this version.
Kind regards,
Denis
On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wg...@gmail.com> wrote:
> Thank you all.
>
> I will be preparing 3.3.1 RC1 soon.
>
> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org> wrote:
>
>> +1
>>
>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com>
>> wrote:
>>
>>> +1
>>>
>>>
>>>
>>> Thanks Yuming ~
>>>
>>>
>>>
>>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>>> *日期**: *2022年9月13日 星期二 08:19
>>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>>> dev@spark.apache.org>
>>> *主题**: *Re: Time for Spark 3.3.1 release?
>>>
>>>
>>>
>>> +1
>>>
>>>
>>>
>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com> wrote:
>>>
>>> +1.
>>>
>>> Thank you, Yuming!
>>>
>>>
>>>
>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com> wrote:
>>>
>>> +1
>>>
>>> Thanks Yuming!
>>>
>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>> >
>>> > +1
>>> >
>>> > Thanks,
>>> > Dongjoon.
>>> >
>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
>>> >>
>>> >> Hi, All.
>>> >>
>>> >>
>>> >>
>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>>> including 7 correctness patches arrived at branch-3.3.
>>> >>
>>> >>
>>> >>
>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second
>>> release at branch-3.3? I'd like to volunteer as the release manager for
>>> Apache Spark 3.3.1.
>>> >>
>>> >>
>>> >>
>>> >> All changes:
>>> >>
>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>> >>
>>> >>
>>> >>
>>> >> Correctness issues:
>>> >>
>>> >> SPARK-40149: Propagate metadata columns through Project
>>> >>
>>> >> SPARK-40002: Don't push down limit through window using ntile
>>> >>
>>> >> SPARK-39976: ArrayIntersect should handle null in left expression
>>> correctly
>>> >>
>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a
>>> correctness issue in the case of overlapping partition and data columns
>>> >>
>>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>>> >>
>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make the
>>> output of projection nodes unique
>>> >>
>>> >> SPARK-38614: Don't push down limit through window that's using
>>> percent_rank
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>> --
>> John Zhuge
>>
>
--
//with Best Regards
--Denis Bolshakov
e-mail: bolshakov.denis@gmail.com
Re: Time for Spark 3.3.1 release?
Posted by Yuming Wang <wg...@gmail.com>.
Thank you all.
I will be preparing 3.3.1 RC1 soon.
On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jz...@apache.org> wrote:
> +1
>
> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com> wrote:
>
>> +1
>>
>>
>>
>> Thanks Yuming ~
>>
>>
>>
>> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
>> *日期**: *2022年9月13日 星期二 08:19
>> *收件人**: *Gengliang Wang <lt...@gmail.com>
>> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
>> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
>> dev@spark.apache.org>
>> *主题**: *Re: Time for Spark 3.3.1 release?
>>
>>
>>
>> +1
>>
>>
>>
>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com> wrote:
>>
>> +1.
>>
>> Thank you, Yuming!
>>
>>
>>
>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com> wrote:
>>
>> +1
>>
>> Thanks Yuming!
>>
>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>> >
>> > +1
>> >
>> > Thanks,
>> > Dongjoon.
>> >
>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
>> >>
>> >> Hi, All.
>> >>
>> >>
>> >>
>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>> including 7 correctness patches arrived at branch-3.3.
>> >>
>> >>
>> >>
>> >> Shall we make a new release, Apache Spark 3.3.1, as the second release
>> at branch-3.3? I'd like to volunteer as the release manager for Apache
>> Spark 3.3.1.
>> >>
>> >>
>> >>
>> >> All changes:
>> >>
>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>> >>
>> >>
>> >>
>> >> Correctness issues:
>> >>
>> >> SPARK-40149: Propagate metadata columns through Project
>> >>
>> >> SPARK-40002: Don't push down limit through window using ntile
>> >>
>> >> SPARK-39976: ArrayIntersect should handle null in left expression
>> correctly
>> >>
>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness
>> issue in the case of overlapping partition and data columns
>> >>
>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>> >>
>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make the
>> output of projection nodes unique
>> >>
>> >> SPARK-38614: Don't push down limit through window that's using
>> percent_rank
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>> --
> John Zhuge
>
Re: Time for Spark 3.3.1 release?
Posted by John Zhuge <jz...@apache.org>.
+1
On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <ya...@baidu.com> wrote:
> +1
>
>
>
> Thanks Yuming ~
>
>
>
> *发件人**: *Hyukjin Kwon <gu...@gmail.com>
> *日期**: *2022年9月13日 星期二 08:19
> *收件人**: *Gengliang Wang <lt...@gmail.com>
> *抄送**: *"L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <
> dongjoon.hyun@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <
> dev@spark.apache.org>
> *主题**: *Re: Time for Spark 3.3.1 release?
>
>
>
> +1
>
>
>
> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com> wrote:
>
> +1.
>
> Thank you, Yuming!
>
>
>
> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com> wrote:
>
> +1
>
> Thanks Yuming!
>
> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
> >
> > +1
> >
> > Thanks,
> > Dongjoon.
> >
> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
> >>
> >> Hi, All.
> >>
> >>
> >>
> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
> including 7 correctness patches arrived at branch-3.3.
> >>
> >>
> >>
> >> Shall we make a new release, Apache Spark 3.3.1, as the second release
> at branch-3.3? I'd like to volunteer as the release manager for Apache
> Spark 3.3.1.
> >>
> >>
> >>
> >> All changes:
> >>
> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
> >>
> >>
> >>
> >> Correctness issues:
> >>
> >> SPARK-40149: Propagate metadata columns through Project
> >>
> >> SPARK-40002: Don't push down limit through window using ntile
> >>
> >> SPARK-39976: ArrayIntersect should handle null in left expression
> correctly
> >>
> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness
> issue in the case of overlapping partition and data columns
> >>
> >> SPARK-39061: Set nullable correctly for Inline output attributes
> >>
> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make the
> output of projection nodes unique
> >>
> >> SPARK-38614: Don't push down limit through window that's using
> percent_rank
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
> --
John Zhuge
Re: Time for Spark 3.3.1 release?
Posted by "Yang,Jie(INF)" <ya...@baidu.com>.
+1
Thanks Yuming ~
发件人: Hyukjin Kwon <gu...@gmail.com>
日期: 2022年9月13日 星期二 08:19
收件人: Gengliang Wang <lt...@gmail.com>
抄送: "L. C. Hsieh" <vi...@gmail.com>, Dongjoon Hyun <do...@gmail.com>, Yuming Wang <wg...@gmail.com>, dev <de...@spark.apache.org>
主题: Re: Time for Spark 3.3.1 release?
+1
On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com>> wrote:
+1.
Thank you, Yuming!
On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com>> wrote:
+1
Thanks Yuming!
On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com>> wrote:
>
> +1
>
> Thanks,
> Dongjoon.
>
> On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com>> wrote:
>>
>> Hi, All.
>>
>>
>>
>> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches including 7 correctness patches arrived at branch-3.3.
>>
>>
>>
>> Shall we make a new release, Apache Spark 3.3.1, as the second release at branch-3.3? I'd like to volunteer as the release manager for Apache Spark 3.3.1.
>>
>>
>>
>> All changes:
>>
>> https://github.com/apache/spark/compare/v3.3.0...branch-3.3<https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d>
>>
>>
>>
>> Correctness issues:
>>
>> SPARK-40149: Propagate metadata columns through Project
>>
>> SPARK-40002: Don't push down limit through window using ntile
>>
>> SPARK-39976: ArrayIntersect should handle null in left expression correctly
>>
>> SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness issue in the case of overlapping partition and data columns
>>
>> SPARK-39061: Set nullable correctly for Inline output attributes
>>
>> SPARK-39887: RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
>>
>> SPARK-38614: Don't push down limit through window that's using percent_rank
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
Re: Time for Spark 3.3.1 release?
Posted by Hyukjin Kwon <gu...@gmail.com>.
+1
On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <lt...@gmail.com> wrote:
> +1.
> Thank you, Yuming!
>
> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com> wrote:
>
>> +1
>>
>> Thanks Yuming!
>>
>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>> >
>> > +1
>> >
>> > Thanks,
>> > Dongjoon.
>> >
>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
>> >>
>> >> Hi, All.
>> >>
>> >>
>> >>
>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
>> including 7 correctness patches arrived at branch-3.3.
>> >>
>> >>
>> >>
>> >> Shall we make a new release, Apache Spark 3.3.1, as the second release
>> at branch-3.3? I'd like to volunteer as the release manager for Apache
>> Spark 3.3.1.
>> >>
>> >>
>> >>
>> >> All changes:
>> >>
>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>> >>
>> >>
>> >>
>> >> Correctness issues:
>> >>
>> >> SPARK-40149: Propagate metadata columns through Project
>> >>
>> >> SPARK-40002: Don't push down limit through window using ntile
>> >>
>> >> SPARK-39976: ArrayIntersect should handle null in left expression
>> correctly
>> >>
>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness
>> issue in the case of overlapping partition and data columns
>> >>
>> >> SPARK-39061: Set nullable correctly for Inline output attributes
>> >>
>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make the
>> output of projection nodes unique
>> >>
>> >> SPARK-38614: Don't push down limit through window that's using
>> percent_rank
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: Time for Spark 3.3.1 release?
Posted by Gengliang Wang <lt...@gmail.com>.
+1.
Thank you, Yuming!
On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vi...@gmail.com> wrote:
> +1
>
> Thanks Yuming!
>
> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
> >
> > +1
> >
> > Thanks,
> > Dongjoon.
> >
> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
> >>
> >> Hi, All.
> >>
> >>
> >>
> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches
> including 7 correctness patches arrived at branch-3.3.
> >>
> >>
> >>
> >> Shall we make a new release, Apache Spark 3.3.1, as the second release
> at branch-3.3? I'd like to volunteer as the release manager for Apache
> Spark 3.3.1.
> >>
> >>
> >>
> >> All changes:
> >>
> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
> >>
> >>
> >>
> >> Correctness issues:
> >>
> >> SPARK-40149: Propagate metadata columns through Project
> >>
> >> SPARK-40002: Don't push down limit through window using ntile
> >>
> >> SPARK-39976: ArrayIntersect should handle null in left expression
> correctly
> >>
> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness
> issue in the case of overlapping partition and data columns
> >>
> >> SPARK-39061: Set nullable correctly for Inline output attributes
> >>
> >> SPARK-39887: RemoveRedundantAliases should keep aliases that make the
> output of projection nodes unique
> >>
> >> SPARK-38614: Don't push down limit through window that's using
> percent_rank
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Time for Spark 3.3.1 release?
Posted by "L. C. Hsieh" <vi...@gmail.com>.
+1
Thanks Yuming!
On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun <do...@gmail.com> wrote:
>
> +1
>
> Thanks,
> Dongjoon.
>
> On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
>>
>> Hi, All.
>>
>>
>>
>> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches including 7 correctness patches arrived at branch-3.3.
>>
>>
>>
>> Shall we make a new release, Apache Spark 3.3.1, as the second release at branch-3.3? I'd like to volunteer as the release manager for Apache Spark 3.3.1.
>>
>>
>>
>> All changes:
>>
>> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>>
>>
>>
>> Correctness issues:
>>
>> SPARK-40149: Propagate metadata columns through Project
>>
>> SPARK-40002: Don't push down limit through window using ntile
>>
>> SPARK-39976: ArrayIntersect should handle null in left expression correctly
>>
>> SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness issue in the case of overlapping partition and data columns
>>
>> SPARK-39061: Set nullable correctly for Inline output attributes
>>
>> SPARK-39887: RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
>>
>> SPARK-38614: Don't push down limit through window that's using percent_rank
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Time for Spark 3.3.1 release?
Posted by Dongjoon Hyun <do...@gmail.com>.
+1
Thanks,
Dongjoon.
On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wg...@gmail.com> wrote:
> Hi, All.
>
>
>
> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches including
> 7 correctness patches arrived at branch-3.3.
>
>
>
> Shall we make a new release, Apache Spark 3.3.1, as the second release at
> branch-3.3? I'd like to volunteer as the release manager for Apache Spark
> 3.3.1.
>
>
>
> All changes:
>
> https://github.com/apache/spark/compare/v3.3.0...branch-3.3
>
>
>
> Correctness issues:
>
> SPARK-40149: Propagate metadata columns through Project
>
> SPARK-40002: Don't push down limit through window using ntile
>
> SPARK-39976: ArrayIntersect should handle null in left expression
> correctly
>
> SPARK-39833: Disable Parquet column index in DSv1 to fix a correctness
> issue in the case of overlapping partition and data columns
>
> SPARK-39061: Set nullable correctly for Inline output attributes
>
> SPARK-39887: RemoveRedundantAliases should keep aliases that make the
> output of projection nodes unique
>
> SPARK-38614: Don't push down limit through window that's using
> percent_rank
>