You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dongjoon Hyun <do...@gmail.com> on 2021/05/17 21:08:46 UTC
Apache Spark 3.1.2 Release?
Hi, All.
Since Apache Spark 3.1.1 tag creation (Feb 21),
new 172 patches including 9 correctness patches and 4 K8s patches arrived
at branch-3.1.
Shall we make a new release, Apache Spark 3.1.2, as the second release at
3.1 line?
I'd like to volunteer for the release manager for Apache Spark 3.1.2.
I'm thinking about starting the first RC next week.
$ git log --oneline v3.1.1..HEAD | wc -l
172
# Known correctness issues
SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
lead to data loss or correctness
SPARK-34545 PySpark Python UDF return inconsistent results when
applying 2 UDFs with different return type to 2 columns together
SPARK-34681 Full outer shuffled hash join when building left side
produces wrong result
SPARK-34719 fail if the view query has duplicated column names
SPARK-34794 Nested higher-order functions broken in DSL
SPARK-34829 transform_values return identical values when it's used
with udf that returns reference type
SPARK-34833 Apply right-padding correctly for correlated subqueries
SPARK-35381 Fix lambda variable name issues in nested DataFrame
functions in R APIs
SPARK-35382 Fix lambda variable name issues in nested DataFrame
functions in Python APIs
# Notable K8s patches since K8s GA
SPARK-34674 Close SparkContext after the Main method has finished
SPARK-34948 Add ownerReference to executor configmap to fix leakages
SPARK-34820 add apt-update before gnupg install
SPARK-34361 In case of downscaling avoid killing of executors already
known by the scheduler backend in the pod allocator
Bests,
Dongjoon.
Re: Apache Spark 3.1.2 Release?
Posted by Gengliang Wang <lt...@gmail.com>.
Late +1, thank you, Dongjoon!
> On May 19, 2021, at 10:47 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Late +1 here as well, thanks for volunteering!
>
> 2021년 5월 19일 (수) 오전 11:24, 郑瑞峰 <ruifengz@foxmail.com <ma...@foxmail.com>>님이 작성:
> late +1. thanks Dongjoon!
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Dongjoon Hyun" <dongjoon.hyun@gmail.com <ma...@gmail.com>>;
> 发送时间: 2021年5月19日(星期三) 凌晨1:29
> 收件人: "Wenchen Fan"<cloud0fan@gmail.com <ma...@gmail.com>>;
> 抄送: "Xiao Li"<lixiao@databricks.com <ma...@databricks.com>>;"Kent Yao"<yaooqinn@gmail.com <ma...@gmail.com>>;"John Zhuge"<jzhuge@apache.org <ma...@apache.org>>;"Hyukjin Kwon"<gurwls223@gmail.com <ma...@gmail.com>>;"Holden Karau"<holden@pigscanfly.ca <ma...@pigscanfly.ca>>;"Takeshi Yamamuro"<linguin.m.s@gmail.com <ma...@gmail.com>>;"dev"<dev@spark.apache.org <ma...@spark.apache.org>>;"Yuming Wang"<wgyumg@gmail.com <ma...@gmail.com>>;
> 主题: Re: Apache Spark 3.1.2 Release?
>
> Thank you all! I'll start to prepare.
>
> Bests,
> Dongjoon.
>
> On Tue, May 18, 2021 at 12:53 AM Wenchen Fan <cloud0fan@gmail.com <ma...@gmail.com>> wrote:
> +1, thanks!
>
> On Tue, May 18, 2021 at 1:37 PM Xiao Li <lixiao@databricks.com <ma...@databricks.com>> wrote:
> +1 Thanks, Dongjoon!
>
> Xiao
>
>
>
> On Mon, May 17, 2021 at 8:45 PM Kent Yao <yaooqinn@gmail.com <ma...@gmail.com>> wrote:
> +1. thanks Dongjoon
>
> Kent Yao
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> a spark enthusiast
> kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark <http://spark.apache.org/>.
> spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL extension which provides SQL Standard Authorization for Apache Spark <http://spark.apache.org/>.
> spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
> itatchi <https://github.com/yaooqinn/spark-func-extras>A library that brings useful functions from various modern database management systems to Apache Spark <http://spark.apache.org/>.
>
>
>
>
> On 05/18/2021 10:57，John Zhuge<jz...@apache.org> <ma...@apache.org> wrote：
> +1, thanks Dongjoon!
>
> On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wgyumg@gmail.com <ma...@gmail.com>> wrote:
> +1.
>
> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gurwls223@gmail.com <ma...@gmail.com>> wrote:
> +1 thanks for driving me
>
> On Tue, 18 May 2021, 09:33 Holden Karau, <holden@pigscanfly.ca <ma...@pigscanfly.ca>> wrote:
> +1 and thanks for volunteering to be the RM :)
>
> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <linguin.m.s@gmail.com <ma...@gmail.com>> wrote:
> Thank you, Dongjoon~ sgtm, too.
>
> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid> wrote:
> +1 for a new release, thanks Dongjoon!
>
> Cheng Su
>
> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <viirya@gmail.com <ma...@gmail.com>> wrote:
>
> +1 sounds good. Thanks Dongjoon for volunteering on this!
>
>
> Liang-Chi
>
>
> Dongjoon Hyun-2 wrote
> > Hi, All.
> >
> > Since Apache Spark 3.1.1 tag creation (Feb 21),
> > new 172 patches including 9 correctness patches and 4 K8s patches arrived
> > at branch-3.1.
> >
> > Shall we make a new release, Apache Spark 3.1.2, as the second release at
> > 3.1 line?
> > I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> > I'm thinking about starting the first RC next week.
> >
> > $ git log --oneline v3.1.1..HEAD | wc -l
> > 172
> >
> > # Known correctness issues
> > SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
> > lead to data loss or correctness
> > SPARK-34545 PySpark Python UDF return inconsistent results when
> > applying 2 UDFs with different return type to 2 columns together
> > SPARK-34681 Full outer shuffled hash join when building left side
> > produces wrong result
> > SPARK-34719 fail if the view query has duplicated column names
> > SPARK-34794 Nested higher-order functions broken in DSL
> > SPARK-34829 transform_values return identical values when it's used
> > with udf that returns reference type
> > SPARK-34833 Apply right-padding correctly for correlated subqueries
> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
> > functions in R APIs
> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
> > functions in Python APIs
> >
> > # Notable K8s patches since K8s GA
> > SPARK-34674 Close SparkContext after the Main method has finished
> > SPARK-34948 Add ownerReference to executor configmap to fix leakages
> > SPARK-34820 add apt-update before gnupg install
> > SPARK-34361 In case of downscaling avoid killing of executors already
> > known by the scheduler backend in the pod allocator
> >
> > Bests,
> > Dongjoon.
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ <http://apache-spark-developers-list.1001551.n3.nabble.com/>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>
>
>
>
> --
> ---
> Takeshi Yamamuro
> --
> Twitter: https://twitter.com/holdenkarau <https://twitter.com/holdenkarau>
> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau <https://www.youtube.com/user/holdenkarau>
>
> --
> John Zhuge
>
>
> --
>
Re: Apache Spark 3.1.2 Release?
Posted by Jungtaek Lim <ka...@gmail.com>.
Late +1 here as well, thanks for volunteering!
2021년 5월 19일 (수) 오전 11:24, 郑瑞峰 <ru...@foxmail.com>님이 작성:
> late +1. thanks Dongjoon!
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Dongjoon Hyun" <do...@gmail.com>;
> *发送时间:* 2021年5月19日(星期三) 凌晨1:29
> *收件人:* "Wenchen Fan"<cl...@gmail.com>;
> *抄送:* "Xiao Li"<li...@databricks.com>;"Kent Yao"<ya...@gmail.com>;"John
> Zhuge"<jz...@apache.org>;"Hyukjin Kwon"<gu...@gmail.com>;"Holden
> Karau"<ho...@pigscanfly.ca>;"Takeshi Yamamuro"<linguin.m.s@gmail.com
> >;"dev"<de...@spark.apache.org>;"Yuming Wang"<wg...@gmail.com>;
> *主题:* Re: Apache Spark 3.1.2 Release?
>
> Thank you all! I'll start to prepare.
>
> Bests,
> Dongjoon.
>
> On Tue, May 18, 2021 at 12:53 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>> +1, thanks!
>>
>> On Tue, May 18, 2021 at 1:37 PM Xiao Li <li...@databricks.com> wrote:
>>
>>> +1 Thanks, Dongjoon!
>>>
>>> Xiao
>>>
>>>
>>>
>>> On Mon, May 17, 2021 at 8:45 PM Kent Yao <ya...@gmail.com> wrote:
>>>
>>>> +1. thanks Dongjoon
>>>>
>>>> *Kent Yao *
>>>> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
>>>> *a spark enthusiast*
>>>> *kyuubi <https://github.com/yaooqinn/kyuubi>is a
>>>> unified multi-tenant JDBC interface for large-scale data processing and
>>>> analytics, built on top of Apache Spark <http://spark.apache.org/>.*
>>>> *spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
>>>> SQL extension which provides SQL Standard Authorization for **Apache
>>>> Spark <http://spark.apache.org/>.*
>>>> *spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
>>>> for reading data from and transferring data to Postgres / Greenplum with
>>>> Spark SQL and DataFrames, 10~100x faster.*
>>>> *itatchi <https://github.com/yaooqinn/spark-func-extras>A** library t**hat
>>>> brings useful functions from various modern database management systems to **Apache
>>>> Spark <http://spark.apache.org/>.*
>>>>
>>>>
>>>>
>>>> On 05/18/2021 10:57，John Zhuge<jz...@apache.org> <jz...@apache.org>
>>>> wrote：
>>>>
>>>> +1, thanks Dongjoon!
>>>>
>>>> On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wg...@gmail.com> wrote:
>>>>
>>>>> +1.
>>>>>
>>>>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1 thanks for driving me
>>>>>>
>>>>>> On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 and thanks for volunteering to be the RM :)
>>>>>>>
>>>>>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <
>>>>>>> linguin.m.s@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thank you, Dongjoon~ sgtm, too.
>>>>>>>>
>>>>>>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 for a new release, thanks Dongjoon!
>>>>>>>>>
>>>>>>>>> Cheng Su
>>>>>>>>>
>>>>>>>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Liang-Chi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dongjoon Hyun-2 wrote
>>>>>>>>> > Hi, All.
>>>>>>>>> >
>>>>>>>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>>>>>>>> > new 172 patches including 9 correctness patches and 4 K8s
>>>>>>>>> patches arrived
>>>>>>>>> > at branch-3.1.
>>>>>>>>> >
>>>>>>>>> > Shall we make a new release, Apache Spark 3.1.2, as the
>>>>>>>>> second release at
>>>>>>>>> > 3.1 line?
>>>>>>>>> > I'd like to volunteer for the release manager for Apache
>>>>>>>>> Spark 3.1.2.
>>>>>>>>> > I'm thinking about starting the first RC next week.
>>>>>>>>> >
>>>>>>>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>>>>>>>> > 172
>>>>>>>>> >
>>>>>>>>> > # Known correctness issues
>>>>>>>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>>>>>>>> OneForOneBlockFetcher
>>>>>>>>> > lead to data loss or correctness
>>>>>>>>> > SPARK-34545 PySpark Python UDF return inconsistent
>>>>>>>>> results when
>>>>>>>>> > applying 2 UDFs with different return type to 2 columns
>>>>>>>>> together
>>>>>>>>> > SPARK-34681 Full outer shuffled hash join when building
>>>>>>>>> left side
>>>>>>>>> > produces wrong result
>>>>>>>>> > SPARK-34719 fail if the view query has duplicated column
>>>>>>>>> names
>>>>>>>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>>>>>>>> > SPARK-34829 transform_values return identical values
>>>>>>>>> when it's used
>>>>>>>>> > with udf that returns reference type
>>>>>>>>> > SPARK-34833 Apply right-padding correctly for correlated
>>>>>>>>> subqueries
>>>>>>>>> > SPARK-35381 Fix lambda variable name issues in nested
>>>>>>>>> DataFrame
>>>>>>>>> > functions in R APIs
>>>>>>>>> > SPARK-35382 Fix lambda variable name issues in nested
>>>>>>>>> DataFrame
>>>>>>>>> > functions in Python APIs
>>>>>>>>> >
>>>>>>>>> > # Notable K8s patches since K8s GA
>>>>>>>>> > SPARK-34674 Close SparkContext after the Main method has
>>>>>>>>> finished
>>>>>>>>> > SPARK-34948 Add ownerReference to executor configmap to
>>>>>>>>> fix leakages
>>>>>>>>> > SPARK-34820 add apt-update before gnupg install
>>>>>>>>> > SPARK-34361 In case of downscaling avoid killing of
>>>>>>>>> executors already
>>>>>>>>> > known by the scheduler backend in the pod allocator
>>>>>>>>> >
>>>>>>>>> > Bests,
>>>>>>>>> > Dongjoon.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Sent from:
>>>>>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> ---
>>>>>>>> Takeshi Yamamuro
>>>>>>>>
>>>>>>> --
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>>>
>>>
>>> --
>>>
>>>
回复： Apache Spark 3.1.2 Release?
Posted by 郑瑞峰 <ru...@foxmail.com>.
late +1. thanks Dongjoon!
------------------ 原始邮件 ------------------
发件人: "Dongjoon Hyun" <dongjoon.hyun@gmail.com>;
发送时间: 2021年5月19日(星期三) 凌晨1:29
收件人: "Wenchen Fan"<cloud0fan@gmail.com>;
抄送: "Xiao Li"<lixiao@databricks.com>;"Kent Yao"<yaooqinn@gmail.com>;"John Zhuge"<jzhuge@apache.org>;"Hyukjin Kwon"<gurwls223@gmail.com>;"Holden Karau"<holden@pigscanfly.ca>;"Takeshi Yamamuro"<linguin.m.s@gmail.com>;"dev"<dev@spark.apache.org>;"Yuming Wang"<wgyumg@gmail.com>;
主题: Re: Apache Spark 3.1.2 Release?
Thank you all! I'll start to prepare.
Bests,
Dongjoon.
On Tue, May 18, 2021 at 12:53 AM Wenchen Fan <cloud0fan@gmail.com> wrote:
+1, thanks!
On Tue, May 18, 2021 at 1:37 PM Xiao Li <lixiao@databricks.com> wrote:
+1 Thanks, Dongjoon!
Xiao
On Mon, May 17, 2021 at 8:45 PM Kent Yao <yaooqinn@gmail.com> wrote:
+1. thanks Dongjoon
Kent Yao
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark.
spark-authorizerA Spark SQL extension which provides SQL Standard Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
itatchiA library that brings useful functions from various modern database management systems to Apache Spark.
On 05/18/2021 10:57，John Zhuge<jzhuge@apache.org> wrote：
+1, thanks Dongjoon!
On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wgyumg@gmail.com> wrote:
+1.
On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gurwls223@gmail.com> wrote:
+1 thanks for driving me
On Tue, 18 May 2021, 09:33 Holden Karau, <holden@pigscanfly.ca> wrote:
+1 and thanks for volunteering to be the RM :)
On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <linguin.m.s@gmail.com> wrote:
Thank you, Dongjoon~ sgtm, too.
On Tue, May 18, 2021 at 7:34 AM Cheng Su <chengsu@fb.com.invalid> wrote:
+1 for a new release, thanks Dongjoon!
Cheng Su
On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <viirya@gmail.com> wrote:
+1 sounds good. Thanks Dongjoon for volunteering on this!
Liang-Chi
Dongjoon Hyun-2 wrote
> Hi, All.
>
> Since Apache Spark 3.1.1 tag creation (Feb 21),
> new 172 patches including 9 correctness patches and 4 K8s patches arrived
> at branch-3.1.
>
> Shall we make a new release, Apache Spark 3.1.2, as the second release at
> 3.1 line?
> I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> I'm thinking about starting the first RC next week.
>
> $ git log --oneline v3.1.1..HEAD | wc -l
> 172
>
> # Known correctness issues
> SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
> lead to data loss or correctness
> SPARK-34545 PySpark Python UDF return inconsistent results when
> applying 2 UDFs with different return type to 2 columns together
> SPARK-34681 Full outer shuffled hash join when building left side
> produces wrong result
> SPARK-34719 fail if the view query has duplicated column names
> SPARK-34794 Nested higher-order functions broken in DSL
> SPARK-34829 transform_values return identical values when it's used
> with udf that returns reference type
> SPARK-34833 Apply right-padding correctly for correlated subqueries
> SPARK-35381 Fix lambda variable name issues in nested DataFrame
> functions in R APIs
> SPARK-35382 Fix lambda variable name issues in nested DataFrame
> functions in Python APIs
>
> # Notable K8s patches since K8s GA
> SPARK-34674 Close SparkContext after the Main method has finished
> SPARK-34948 Add ownerReference to executor configmap to fix leakages
> SPARK-34820 add apt-update before gnupg install
> SPARK-34361 In case of downscaling avoid killing of executors already
> known by the scheduler backend in the pod allocator
>
> Bests,
> Dongjoon.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
--
---
Takeshi Yamamuro
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
--
John Zhuge
--
Re: Apache Spark 3.1.2 Release?
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you all! I'll start to prepare.
Bests,
Dongjoon.
On Tue, May 18, 2021 at 12:53 AM Wenchen Fan <cl...@gmail.com> wrote:
> +1, thanks!
>
> On Tue, May 18, 2021 at 1:37 PM Xiao Li <li...@databricks.com> wrote:
>
>> +1 Thanks, Dongjoon!
>>
>> Xiao
>>
>>
>>
>> On Mon, May 17, 2021 at 8:45 PM Kent Yao <ya...@gmail.com> wrote:
>>
>>> +1. thanks Dongjoon
>>>
>>> *Kent Yao *
>>> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
>>> *a spark enthusiast*
>>> *kyuubi <https://github.com/yaooqinn/kyuubi>is a
>>> unified multi-tenant JDBC interface for large-scale data processing and
>>> analytics, built on top of Apache Spark <http://spark.apache.org/>.*
>>> *spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
>>> SQL extension which provides SQL Standard Authorization for **Apache
>>> Spark <http://spark.apache.org/>.*
>>> *spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
>>> for reading data from and transferring data to Postgres / Greenplum with
>>> Spark SQL and DataFrames, 10~100x faster.*
>>> *itatchi <https://github.com/yaooqinn/spark-func-extras>A** library t**hat
>>> brings useful functions from various modern database management systems to **Apache
>>> Spark <http://spark.apache.org/>.*
>>>
>>>
>>>
>>> On 05/18/2021 10:57，John Zhuge<jz...@apache.org> <jz...@apache.org>
>>> wrote：
>>>
>>> +1, thanks Dongjoon!
>>>
>>> On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wg...@gmail.com> wrote:
>>>
>>>> +1.
>>>>
>>>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gu...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 thanks for driving me
>>>>>
>>>>> On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca> wrote:
>>>>>
>>>>>> +1 and thanks for volunteering to be the RM :)
>>>>>>
>>>>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <
>>>>>> linguin.m.s@gmail.com> wrote:
>>>>>>
>>>>>>> Thank you, Dongjoon~ sgtm, too.
>>>>>>>
>>>>>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 for a new release, thanks Dongjoon!
>>>>>>>>
>>>>>>>> Cheng Su
>>>>>>>>
>>>>>>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>>>>>>
>>>>>>>>
>>>>>>>> Liang-Chi
>>>>>>>>
>>>>>>>>
>>>>>>>> Dongjoon Hyun-2 wrote
>>>>>>>> > Hi, All.
>>>>>>>> >
>>>>>>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>>>>>>> > new 172 patches including 9 correctness patches and 4 K8s
>>>>>>>> patches arrived
>>>>>>>> > at branch-3.1.
>>>>>>>> >
>>>>>>>> > Shall we make a new release, Apache Spark 3.1.2, as the
>>>>>>>> second release at
>>>>>>>> > 3.1 line?
>>>>>>>> > I'd like to volunteer for the release manager for Apache
>>>>>>>> Spark 3.1.2.
>>>>>>>> > I'm thinking about starting the first RC next week.
>>>>>>>> >
>>>>>>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>>>>>>> > 172
>>>>>>>> >
>>>>>>>> > # Known correctness issues
>>>>>>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>>>>>>> OneForOneBlockFetcher
>>>>>>>> > lead to data loss or correctness
>>>>>>>> > SPARK-34545 PySpark Python UDF return inconsistent
>>>>>>>> results when
>>>>>>>> > applying 2 UDFs with different return type to 2 columns
>>>>>>>> together
>>>>>>>> > SPARK-34681 Full outer shuffled hash join when building
>>>>>>>> left side
>>>>>>>> > produces wrong result
>>>>>>>> > SPARK-34719 fail if the view query has duplicated column
>>>>>>>> names
>>>>>>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>>>>>>> > SPARK-34829 transform_values return identical values when
>>>>>>>> it's used
>>>>>>>> > with udf that returns reference type
>>>>>>>> > SPARK-34833 Apply right-padding correctly for correlated
>>>>>>>> subqueries
>>>>>>>> > SPARK-35381 Fix lambda variable name issues in nested
>>>>>>>> DataFrame
>>>>>>>> > functions in R APIs
>>>>>>>> > SPARK-35382 Fix lambda variable name issues in nested
>>>>>>>> DataFrame
>>>>>>>> > functions in Python APIs
>>>>>>>> >
>>>>>>>> > # Notable K8s patches since K8s GA
>>>>>>>> > SPARK-34674 Close SparkContext after the Main method has
>>>>>>>> finished
>>>>>>>> > SPARK-34948 Add ownerReference to executor configmap to
>>>>>>>> fix leakages
>>>>>>>> > SPARK-34820 add apt-update before gnupg install
>>>>>>>> > SPARK-34361 In case of downscaling avoid killing of
>>>>>>>> executors already
>>>>>>>> > known by the scheduler backend in the pod allocator
>>>>>>>> >
>>>>>>>> > Bests,
>>>>>>>> > Dongjoon.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sent from:
>>>>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ---
>>>>>>> Takeshi Yamamuro
>>>>>>>
>>>>>> --
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>
>>>>>
>>>
>>> --
>>> John Zhuge
>>>
>>>
>>
>> --
>>
>>
Re: Apache Spark 3.1.2 Release?
Posted by Wenchen Fan <cl...@gmail.com>.
+1, thanks!
On Tue, May 18, 2021 at 1:37 PM Xiao Li <li...@databricks.com> wrote:
> +1 Thanks, Dongjoon!
>
> Xiao
>
>
>
> On Mon, May 17, 2021 at 8:45 PM Kent Yao <ya...@gmail.com> wrote:
>
>> +1. thanks Dongjoon
>>
>> *Kent Yao *
>> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
>> *a spark enthusiast*
>> *kyuubi <https://github.com/yaooqinn/kyuubi>is a
>> unified multi-tenant JDBC interface for large-scale data processing and
>> analytics, built on top of Apache Spark <http://spark.apache.org/>.*
>> *spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
>> SQL extension which provides SQL Standard Authorization for **Apache
>> Spark <http://spark.apache.org/>.*
>> *spark-postgres <https://github.com/yaooqinn/spark-postgres> A library
>> for reading data from and transferring data to Postgres / Greenplum with
>> Spark SQL and DataFrames, 10~100x faster.*
>> *itatchi <https://github.com/yaooqinn/spark-func-extras>A** library t**hat
>> brings useful functions from various modern database management systems to **Apache
>> Spark <http://spark.apache.org/>.*
>>
>>
>>
>> On 05/18/2021 10:57，John Zhuge<jz...@apache.org> <jz...@apache.org>
>> wrote：
>>
>> +1, thanks Dongjoon!
>>
>> On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wg...@gmail.com> wrote:
>>
>>> +1.
>>>
>>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> +1 thanks for driving me
>>>>
>>>> On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca> wrote:
>>>>
>>>>> +1 and thanks for volunteering to be the RM :)
>>>>>
>>>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <
>>>>> linguin.m.s@gmail.com> wrote:
>>>>>
>>>>>> Thank you, Dongjoon~ sgtm, too.
>>>>>>
>>>>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 for a new release, thanks Dongjoon!
>>>>>>>
>>>>>>> Cheng Su
>>>>>>>
>>>>>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>>>>>
>>>>>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>>>>>
>>>>>>>
>>>>>>> Liang-Chi
>>>>>>>
>>>>>>>
>>>>>>> Dongjoon Hyun-2 wrote
>>>>>>> > Hi, All.
>>>>>>> >
>>>>>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>>>>>> > new 172 patches including 9 correctness patches and 4 K8s
>>>>>>> patches arrived
>>>>>>> > at branch-3.1.
>>>>>>> >
>>>>>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>>>>>> release at
>>>>>>> > 3.1 line?
>>>>>>> > I'd like to volunteer for the release manager for Apache Spark
>>>>>>> 3.1.2.
>>>>>>> > I'm thinking about starting the first RC next week.
>>>>>>> >
>>>>>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>>>>>> > 172
>>>>>>> >
>>>>>>> > # Known correctness issues
>>>>>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>>>>>> OneForOneBlockFetcher
>>>>>>> > lead to data loss or correctness
>>>>>>> > SPARK-34545 PySpark Python UDF return inconsistent results
>>>>>>> when
>>>>>>> > applying 2 UDFs with different return type to 2 columns
>>>>>>> together
>>>>>>> > SPARK-34681 Full outer shuffled hash join when building
>>>>>>> left side
>>>>>>> > produces wrong result
>>>>>>> > SPARK-34719 fail if the view query has duplicated column
>>>>>>> names
>>>>>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>>>>>> > SPARK-34829 transform_values return identical values when
>>>>>>> it's used
>>>>>>> > with udf that returns reference type
>>>>>>> > SPARK-34833 Apply right-padding correctly for correlated
>>>>>>> subqueries
>>>>>>> > SPARK-35381 Fix lambda variable name issues in nested
>>>>>>> DataFrame
>>>>>>> > functions in R APIs
>>>>>>> > SPARK-35382 Fix lambda variable name issues in nested
>>>>>>> DataFrame
>>>>>>> > functions in Python APIs
>>>>>>> >
>>>>>>> > # Notable K8s patches since K8s GA
>>>>>>> > SPARK-34674 Close SparkContext after the Main method has
>>>>>>> finished
>>>>>>> > SPARK-34948 Add ownerReference to executor configmap to fix
>>>>>>> leakages
>>>>>>> > SPARK-34820 add apt-update before gnupg install
>>>>>>> > SPARK-34361 In case of downscaling avoid killing of
>>>>>>> executors already
>>>>>>> > known by the scheduler backend in the pod allocator
>>>>>>> >
>>>>>>> > Bests,
>>>>>>> > Dongjoon.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sent from:
>>>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> ---
>>>>>> Takeshi Yamamuro
>>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>
>>>>
>>
>> --
>> John Zhuge
>>
>>
>
> --
>
>
Re: Apache Spark 3.1.2 Release?
Posted by Xiao Li <li...@databricks.com>.
+1 Thanks, Dongjoon!
Xiao
On Mon, May 17, 2021 at 8:45 PM Kent Yao <ya...@gmail.com> wrote:
> +1. thanks Dongjoon
>
> *Kent Yao *
> @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> *a spark enthusiast*
> *kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
> interface for large-scale data processing and analytics, built on top
> of Apache Spark <http://spark.apache.org/>.*
> *spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark
> SQL extension which provides SQL Standard Authorization for **Apache
> Spark <http://spark.apache.org/>.*
> *spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
> reading data from and transferring data to Postgres / Greenplum with Spark
> SQL and DataFrames, 10~100x faster.*
> *itatchi <https://github.com/yaooqinn/spark-func-extras>A** library t**hat
> brings useful functions from various modern database management systems to **Apache
> Spark <http://spark.apache.org/>.*
>
>
>
> On 05/18/2021 10:57，John Zhuge<jz...@apache.org> <jz...@apache.org>
> wrote：
>
> +1, thanks Dongjoon!
>
> On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wg...@gmail.com> wrote:
>
>> +1.
>>
>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> +1 thanks for driving me
>>>
>>> On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca> wrote:
>>>
>>>> +1 and thanks for volunteering to be the RM :)
>>>>
>>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <li...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thank you, Dongjoon~ sgtm, too.
>>>>>
>>>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> +1 for a new release, thanks Dongjoon!
>>>>>>
>>>>>> Cheng Su
>>>>>>
>>>>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>>>>
>>>>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>>>>
>>>>>>
>>>>>> Liang-Chi
>>>>>>
>>>>>>
>>>>>> Dongjoon Hyun-2 wrote
>>>>>> > Hi, All.
>>>>>> >
>>>>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>>>>> > new 172 patches including 9 correctness patches and 4 K8s
>>>>>> patches arrived
>>>>>> > at branch-3.1.
>>>>>> >
>>>>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>>>>> release at
>>>>>> > 3.1 line?
>>>>>> > I'd like to volunteer for the release manager for Apache Spark
>>>>>> 3.1.2.
>>>>>> > I'm thinking about starting the first RC next week.
>>>>>> >
>>>>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>>>>> > 172
>>>>>> >
>>>>>> > # Known correctness issues
>>>>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>>>>> OneForOneBlockFetcher
>>>>>> > lead to data loss or correctness
>>>>>> > SPARK-34545 PySpark Python UDF return inconsistent results
>>>>>> when
>>>>>> > applying 2 UDFs with different return type to 2 columns together
>>>>>> > SPARK-34681 Full outer shuffled hash join when building
>>>>>> left side
>>>>>> > produces wrong result
>>>>>> > SPARK-34719 fail if the view query has duplicated column
>>>>>> names
>>>>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>>>>> > SPARK-34829 transform_values return identical values when
>>>>>> it's used
>>>>>> > with udf that returns reference type
>>>>>> > SPARK-34833 Apply right-padding correctly for correlated
>>>>>> subqueries
>>>>>> > SPARK-35381 Fix lambda variable name issues in nested
>>>>>> DataFrame
>>>>>> > functions in R APIs
>>>>>> > SPARK-35382 Fix lambda variable name issues in nested
>>>>>> DataFrame
>>>>>> > functions in Python APIs
>>>>>> >
>>>>>> > # Notable K8s patches since K8s GA
>>>>>> > SPARK-34674 Close SparkContext after the Main method has
>>>>>> finished
>>>>>> > SPARK-34948 Add ownerReference to executor configmap to fix
>>>>>> leakages
>>>>>> > SPARK-34820 add apt-update before gnupg install
>>>>>> > SPARK-34361 In case of downscaling avoid killing of
>>>>>> executors already
>>>>>> > known by the scheduler backend in the pod allocator
>>>>>> >
>>>>>> > Bests,
>>>>>> > Dongjoon.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sent from:
>>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> ---
>>>>> Takeshi Yamamuro
>>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>
>
> --
> John Zhuge
>
>
--
Re: Apache Spark 3.1.2 Release?
Posted by Kent Yao <ya...@gmail.com>.
+1. thanks Dongjoon
**Kent Yao **
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
** _a s_ _park enthusiast_**
_[kyuubi](https://github.com/yaooqinn/kyuubi) is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of
[Apache Spark](http://spark.apache.org/)._
#
_[spark-authorizer](https://github.com/yaooqinn/spark-authorizer) A Spark SQL
extension which provides SQL Standard Authorization for _ _[Apache
Spark](http://spark.apache.org/)._
_ _[spark-postgres](https://github.com/yaooqinn/spark-postgres) A library
for reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.__
_ _[itatchi](https://github.com/yaooqinn/spark-func-extras) A__ _library t_
_hat brings useful functions from various modern database management systems
to _ _ _[Apache Spark](http://spark.apache.org/).__
#
On 05/18/2021 10:57，[John Zhuge<jz...@apache.org>](mailto:jzhuge@apache.org)
wrote：
> +1, thanks Dongjoon!
>
>
>
>
>
> On Mon, May 17, 2021 at 7:50 PM Yuming Wang
<[wgyumg@gmail.com](mailto:wgyumg@gmail.com)> wrote:
>
>
>> +1.
>
>>
>>
>
>>
>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon
<[gurwls223@gmail.com](mailto:gurwls223@gmail.com)> wrote:
>
>>
>>> +1 thanks for driving me
>>>
>>>
>
>>>
>>> On Tue, 18 May 2021, 09:33 Holden Karau,
<[holden@pigscanfly.ca](mailto:holden@pigscanfly.ca)> wrote:
>
>>>
>>>> +1 and thanks for volunteering to be the RM :)
>>>>
>>>>
>
>>>>
>>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro
<[linguin.m.s@gmail.com](mailto:linguin.m.s@gmail.com)> wrote:
>
>>>>
>>>>> Thank you, Dongjoon~ sgtm, too.
>>>>>
>>>>>
>
>>>>>
>>>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid> wrote:
>
>>>>>
>>>>>> +1 for a new release, thanks Dongjoon!
>
> Cheng Su
>
> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh"
<[viirya@gmail.com](mailto:viirya@gmail.com)> wrote:
>
> +1 sounds good. Thanks Dongjoon for volunteering on this!
>
>
> Liang-Chi
>
>
> Dongjoon Hyun-2 wrote
> > Hi, All.
> >
> > Since Apache Spark 3.1.1 tag creation (Feb 21),
> > new 172 patches including 9 correctness patches and 4 K8s patches
arrived
> > at branch-3.1.
> >
> > Shall we make a new release, Apache Spark 3.1.2, as the second
release at
> > 3.1 line?
> > I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> > I'm thinking about starting the first RC next week.
> >
> > $ git log --oneline v3.1.1..HEAD | wc -l
> > 172
> >
> > # Known correctness issues
> > SPARK-34534 New protocol FetchShuffleBlocks in
OneForOneBlockFetcher
> > lead to data loss or correctness
> > SPARK-34545 PySpark Python UDF return inconsistent results when
> > applying 2 UDFs with different return type to 2 columns together
> > SPARK-34681 Full outer shuffled hash join when building left side
> > produces wrong result
> > SPARK-34719 fail if the view query has duplicated column names
> > SPARK-34794 Nested higher-order functions broken in DSL
> > SPARK-34829 transform_values return identical values when it's
used
> > with udf that returns reference type
> > SPARK-34833 Apply right-padding correctly for correlated
subqueries
> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
> > functions in R APIs
> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
> > functions in Python APIs
> >
> > # Notable K8s patches since K8s GA
> > SPARK-34674 Close SparkContext after the Main method has finished
> > SPARK-34948 Add ownerReference to executor configmap to fix
leakages
> > SPARK-34820 add apt-update before gnupg install
> > SPARK-34361 In case of downscaling avoid killing of executors
already
> > known by the scheduler backend in the pod allocator
> >
> > Bests,
> > Dongjoon.
>
>
>
>
>
> \--
> Sent from: <http://apache-spark-developers-list.1001551.n3.nabble.com/>
>
> \---------------------------------------------------------------------
> To unsubscribe e-mail: [dev-unsubscribe@spark.apache.org](mailto:dev-
unsubscribe@spark.apache.org)
>
>
>
>>>>>
>>>>>
>
>>>>>
>>>>>
>
>>>>>
>>>>> \--
>
>>>>>
>>>>> \---
> Takeshi Yamamuro
>
>>>>
>>>> \--
>
>>>>
>>>> Twitter: <https://twitter.com/holdenkarau>
>
>>>>
>>>> Books (Learning Spark, High Performance Spark, etc.):
<https://amzn.to/2MaRAG9>[ ](https://amzn.to/2MaRAG9)
>>>>
>>>> YouTube Live Streams: <https://www.youtube.com/user/holdenkarau>
>
>
>
>
>
>
>
> \--
>
>
> John Zhuge
Re: Apache Spark 3.1.2 Release?
Posted by Chao Sun <su...@apache.org>.
+1. Thanks Dongjoon for doing this!
On Mon, May 17, 2021 at 7:58 PM John Zhuge <jz...@apache.org> wrote:
> +1, thanks Dongjoon!
>
> On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wg...@gmail.com> wrote:
>
>> +1.
>>
>> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> +1 thanks for driving me
>>>
>>> On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca> wrote:
>>>
>>>> +1 and thanks for volunteering to be the RM :)
>>>>
>>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <li...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thank you, Dongjoon~ sgtm, too.
>>>>>
>>>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> +1 for a new release, thanks Dongjoon!
>>>>>>
>>>>>> Cheng Su
>>>>>>
>>>>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>>>>
>>>>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>>>>
>>>>>>
>>>>>> Liang-Chi
>>>>>>
>>>>>>
>>>>>> Dongjoon Hyun-2 wrote
>>>>>> > Hi, All.
>>>>>> >
>>>>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>>>>> > new 172 patches including 9 correctness patches and 4 K8s
>>>>>> patches arrived
>>>>>> > at branch-3.1.
>>>>>> >
>>>>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>>>>> release at
>>>>>> > 3.1 line?
>>>>>> > I'd like to volunteer for the release manager for Apache Spark
>>>>>> 3.1.2.
>>>>>> > I'm thinking about starting the first RC next week.
>>>>>> >
>>>>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>>>>> > 172
>>>>>> >
>>>>>> > # Known correctness issues
>>>>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>>>>> OneForOneBlockFetcher
>>>>>> > lead to data loss or correctness
>>>>>> > SPARK-34545 PySpark Python UDF return inconsistent results
>>>>>> when
>>>>>> > applying 2 UDFs with different return type to 2 columns together
>>>>>> > SPARK-34681 Full outer shuffled hash join when building
>>>>>> left side
>>>>>> > produces wrong result
>>>>>> > SPARK-34719 fail if the view query has duplicated column
>>>>>> names
>>>>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>>>>> > SPARK-34829 transform_values return identical values when
>>>>>> it's used
>>>>>> > with udf that returns reference type
>>>>>> > SPARK-34833 Apply right-padding correctly for correlated
>>>>>> subqueries
>>>>>> > SPARK-35381 Fix lambda variable name issues in nested
>>>>>> DataFrame
>>>>>> > functions in R APIs
>>>>>> > SPARK-35382 Fix lambda variable name issues in nested
>>>>>> DataFrame
>>>>>> > functions in Python APIs
>>>>>> >
>>>>>> > # Notable K8s patches since K8s GA
>>>>>> > SPARK-34674 Close SparkContext after the Main method has
>>>>>> finished
>>>>>> > SPARK-34948 Add ownerReference to executor configmap to fix
>>>>>> leakages
>>>>>> > SPARK-34820 add apt-update before gnupg install
>>>>>> > SPARK-34361 In case of downscaling avoid killing of
>>>>>> executors already
>>>>>> > known by the scheduler backend in the pod allocator
>>>>>> >
>>>>>> > Bests,
>>>>>> > Dongjoon.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sent from:
>>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> ---
>>>>> Takeshi Yamamuro
>>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>
>
> --
> John Zhuge
>
Re: Apache Spark 3.1.2 Release?
Posted by John Zhuge <jz...@apache.org>.
+1, thanks Dongjoon!
On Mon, May 17, 2021 at 7:50 PM Yuming Wang <wg...@gmail.com> wrote:
> +1.
>
> On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> +1 thanks for driving me
>>
>> On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca> wrote:
>>
>>> +1 and thanks for volunteering to be the RM :)
>>>
>>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <li...@gmail.com>
>>> wrote:
>>>
>>>> Thank you, Dongjoon~ sgtm, too.
>>>>
>>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid>
>>>> wrote:
>>>>
>>>>> +1 for a new release, thanks Dongjoon!
>>>>>
>>>>> Cheng Su
>>>>>
>>>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>>>
>>>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>>>
>>>>>
>>>>> Liang-Chi
>>>>>
>>>>>
>>>>> Dongjoon Hyun-2 wrote
>>>>> > Hi, All.
>>>>> >
>>>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>>>> > new 172 patches including 9 correctness patches and 4 K8s
>>>>> patches arrived
>>>>> > at branch-3.1.
>>>>> >
>>>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>>>> release at
>>>>> > 3.1 line?
>>>>> > I'd like to volunteer for the release manager for Apache Spark
>>>>> 3.1.2.
>>>>> > I'm thinking about starting the first RC next week.
>>>>> >
>>>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>>>> > 172
>>>>> >
>>>>> > # Known correctness issues
>>>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>>>> OneForOneBlockFetcher
>>>>> > lead to data loss or correctness
>>>>> > SPARK-34545 PySpark Python UDF return inconsistent results
>>>>> when
>>>>> > applying 2 UDFs with different return type to 2 columns together
>>>>> > SPARK-34681 Full outer shuffled hash join when building left
>>>>> side
>>>>> > produces wrong result
>>>>> > SPARK-34719 fail if the view query has duplicated column
>>>>> names
>>>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>>>> > SPARK-34829 transform_values return identical values when
>>>>> it's used
>>>>> > with udf that returns reference type
>>>>> > SPARK-34833 Apply right-padding correctly for correlated
>>>>> subqueries
>>>>> > SPARK-35381 Fix lambda variable name issues in nested
>>>>> DataFrame
>>>>> > functions in R APIs
>>>>> > SPARK-35382 Fix lambda variable name issues in nested
>>>>> DataFrame
>>>>> > functions in Python APIs
>>>>> >
>>>>> > # Notable K8s patches since K8s GA
>>>>> > SPARK-34674 Close SparkContext after the Main method has
>>>>> finished
>>>>> > SPARK-34948 Add ownerReference to executor configmap to fix
>>>>> leakages
>>>>> > SPARK-34820 add apt-update before gnupg install
>>>>> > SPARK-34361 In case of downscaling avoid killing of executors
>>>>> already
>>>>> > known by the scheduler backend in the pod allocator
>>>>> >
>>>>> > Bests,
>>>>> > Dongjoon.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from:
>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> ---
>>>> Takeshi Yamamuro
>>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
--
John Zhuge
Re: Apache Spark 3.1.2 Release?
Posted by Yuming Wang <wg...@gmail.com>.
+1.
On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon <gu...@gmail.com> wrote:
> +1 thanks for driving me
>
> On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca> wrote:
>
>> +1 and thanks for volunteering to be the RM :)
>>
>> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <li...@gmail.com>
>> wrote:
>>
>>> Thank you, Dongjoon~ sgtm, too.
>>>
>>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid> wrote:
>>>
>>>> +1 for a new release, thanks Dongjoon!
>>>>
>>>> Cheng Su
>>>>
>>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>>
>>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>>
>>>>
>>>> Liang-Chi
>>>>
>>>>
>>>> Dongjoon Hyun-2 wrote
>>>> > Hi, All.
>>>> >
>>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>>> > new 172 patches including 9 correctness patches and 4 K8s patches
>>>> arrived
>>>> > at branch-3.1.
>>>> >
>>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>>> release at
>>>> > 3.1 line?
>>>> > I'd like to volunteer for the release manager for Apache Spark
>>>> 3.1.2.
>>>> > I'm thinking about starting the first RC next week.
>>>> >
>>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>>> > 172
>>>> >
>>>> > # Known correctness issues
>>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>>> OneForOneBlockFetcher
>>>> > lead to data loss or correctness
>>>> > SPARK-34545 PySpark Python UDF return inconsistent results
>>>> when
>>>> > applying 2 UDFs with different return type to 2 columns together
>>>> > SPARK-34681 Full outer shuffled hash join when building left
>>>> side
>>>> > produces wrong result
>>>> > SPARK-34719 fail if the view query has duplicated column names
>>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>>> > SPARK-34829 transform_values return identical values when
>>>> it's used
>>>> > with udf that returns reference type
>>>> > SPARK-34833 Apply right-padding correctly for correlated
>>>> subqueries
>>>> > SPARK-35381 Fix lambda variable name issues in nested
>>>> DataFrame
>>>> > functions in R APIs
>>>> > SPARK-35382 Fix lambda variable name issues in nested
>>>> DataFrame
>>>> > functions in Python APIs
>>>> >
>>>> > # Notable K8s patches since K8s GA
>>>> > SPARK-34674 Close SparkContext after the Main method has
>>>> finished
>>>> > SPARK-34948 Add ownerReference to executor configmap to fix
>>>> leakages
>>>> > SPARK-34820 add apt-update before gnupg install
>>>> > SPARK-34361 In case of downscaling avoid killing of executors
>>>> already
>>>> > known by the scheduler backend in the pod allocator
>>>> >
>>>> > Bests,
>>>> > Dongjoon.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from:
>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
Re: Apache Spark 3.1.2 Release?
Posted by Hyukjin Kwon <gu...@gmail.com>.
+1 thanks for driving me
On Tue, 18 May 2021, 09:33 Holden Karau, <ho...@pigscanfly.ca> wrote:
> +1 and thanks for volunteering to be the RM :)
>
> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <li...@gmail.com>
> wrote:
>
>> Thank you, Dongjoon~ sgtm, too.
>>
>> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid> wrote:
>>
>>> +1 for a new release, thanks Dongjoon!
>>>
>>> Cheng Su
>>>
>>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>>
>>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>>
>>>
>>> Liang-Chi
>>>
>>>
>>> Dongjoon Hyun-2 wrote
>>> > Hi, All.
>>> >
>>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>>> > new 172 patches including 9 correctness patches and 4 K8s patches
>>> arrived
>>> > at branch-3.1.
>>> >
>>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>>> release at
>>> > 3.1 line?
>>> > I'd like to volunteer for the release manager for Apache Spark
>>> 3.1.2.
>>> > I'm thinking about starting the first RC next week.
>>> >
>>> > $ git log --oneline v3.1.1..HEAD | wc -l
>>> > 172
>>> >
>>> > # Known correctness issues
>>> > SPARK-34534 New protocol FetchShuffleBlocks in
>>> OneForOneBlockFetcher
>>> > lead to data loss or correctness
>>> > SPARK-34545 PySpark Python UDF return inconsistent results when
>>> > applying 2 UDFs with different return type to 2 columns together
>>> > SPARK-34681 Full outer shuffled hash join when building left
>>> side
>>> > produces wrong result
>>> > SPARK-34719 fail if the view query has duplicated column names
>>> > SPARK-34794 Nested higher-order functions broken in DSL
>>> > SPARK-34829 transform_values return identical values when it's
>>> used
>>> > with udf that returns reference type
>>> > SPARK-34833 Apply right-padding correctly for correlated
>>> subqueries
>>> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
>>> > functions in R APIs
>>> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
>>> > functions in Python APIs
>>> >
>>> > # Notable K8s patches since K8s GA
>>> > SPARK-34674 Close SparkContext after the Main method has
>>> finished
>>> > SPARK-34948 Add ownerReference to executor configmap to fix
>>> leakages
>>> > SPARK-34820 add apt-update before gnupg install
>>> > SPARK-34361 In case of downscaling avoid killing of executors
>>> already
>>> > known by the scheduler backend in the pod allocator
>>> >
>>> > Bests,
>>> > Dongjoon.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
Re: Apache Spark 3.1.2 Release?
Posted by Holden Karau <ho...@pigscanfly.ca>.
+1 and thanks for volunteering to be the RM :)
On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro <li...@gmail.com>
wrote:
> Thank you, Dongjoon~ sgtm, too.
>
> On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid> wrote:
>
>> +1 for a new release, thanks Dongjoon!
>>
>> Cheng Su
>>
>> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>>
>> +1 sounds good. Thanks Dongjoon for volunteering on this!
>>
>>
>> Liang-Chi
>>
>>
>> Dongjoon Hyun-2 wrote
>> > Hi, All.
>> >
>> > Since Apache Spark 3.1.1 tag creation (Feb 21),
>> > new 172 patches including 9 correctness patches and 4 K8s patches
>> arrived
>> > at branch-3.1.
>> >
>> > Shall we make a new release, Apache Spark 3.1.2, as the second
>> release at
>> > 3.1 line?
>> > I'd like to volunteer for the release manager for Apache Spark
>> 3.1.2.
>> > I'm thinking about starting the first RC next week.
>> >
>> > $ git log --oneline v3.1.1..HEAD | wc -l
>> > 172
>> >
>> > # Known correctness issues
>> > SPARK-34534 New protocol FetchShuffleBlocks in
>> OneForOneBlockFetcher
>> > lead to data loss or correctness
>> > SPARK-34545 PySpark Python UDF return inconsistent results when
>> > applying 2 UDFs with different return type to 2 columns together
>> > SPARK-34681 Full outer shuffled hash join when building left
>> side
>> > produces wrong result
>> > SPARK-34719 fail if the view query has duplicated column names
>> > SPARK-34794 Nested higher-order functions broken in DSL
>> > SPARK-34829 transform_values return identical values when it's
>> used
>> > with udf that returns reference type
>> > SPARK-34833 Apply right-padding correctly for correlated
>> subqueries
>> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
>> > functions in R APIs
>> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
>> > functions in Python APIs
>> >
>> > # Notable K8s patches since K8s GA
>> > SPARK-34674 Close SparkContext after the Main method has finished
>> > SPARK-34948 Add ownerReference to executor configmap to fix
>> leakages
>> > SPARK-34820 add apt-update before gnupg install
>> > SPARK-34361 In case of downscaling avoid killing of executors
>> already
>> > known by the scheduler backend in the pod allocator
>> >
>> > Bests,
>> > Dongjoon.
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: Apache Spark 3.1.2 Release?
Posted by Takeshi Yamamuro <li...@gmail.com>.
Thank you, Dongjoon~ sgtm, too.
On Tue, May 18, 2021 at 7:34 AM Cheng Su <ch...@fb.com.invalid> wrote:
> +1 for a new release, thanks Dongjoon!
>
> Cheng Su
>
> On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
>
> +1 sounds good. Thanks Dongjoon for volunteering on this!
>
>
> Liang-Chi
>
>
> Dongjoon Hyun-2 wrote
> > Hi, All.
> >
> > Since Apache Spark 3.1.1 tag creation (Feb 21),
> > new 172 patches including 9 correctness patches and 4 K8s patches
> arrived
> > at branch-3.1.
> >
> > Shall we make a new release, Apache Spark 3.1.2, as the second
> release at
> > 3.1 line?
> > I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> > I'm thinking about starting the first RC next week.
> >
> > $ git log --oneline v3.1.1..HEAD | wc -l
> > 172
> >
> > # Known correctness issues
> > SPARK-34534 New protocol FetchShuffleBlocks in
> OneForOneBlockFetcher
> > lead to data loss or correctness
> > SPARK-34545 PySpark Python UDF return inconsistent results when
> > applying 2 UDFs with different return type to 2 columns together
> > SPARK-34681 Full outer shuffled hash join when building left side
> > produces wrong result
> > SPARK-34719 fail if the view query has duplicated column names
> > SPARK-34794 Nested higher-order functions broken in DSL
> > SPARK-34829 transform_values return identical values when it's
> used
> > with udf that returns reference type
> > SPARK-34833 Apply right-padding correctly for correlated
> subqueries
> > SPARK-35381 Fix lambda variable name issues in nested DataFrame
> > functions in R APIs
> > SPARK-35382 Fix lambda variable name issues in nested DataFrame
> > functions in Python APIs
> >
> > # Notable K8s patches since K8s GA
> > SPARK-34674 Close SparkContext after the Main method has finished
> > SPARK-34948 Add ownerReference to executor configmap to fix
> leakages
> > SPARK-34820 add apt-update before gnupg install
> > SPARK-34361 In case of downscaling avoid killing of executors
> already
> > known by the scheduler backend in the pod allocator
> >
> > Bests,
> > Dongjoon.
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
--
---
Takeshi Yamamuro
Re: Apache Spark 3.1.2 Release?
Posted by Cheng Su <ch...@fb.com.INVALID>.
+1 for a new release, thanks Dongjoon!
Cheng Su
On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" <vi...@gmail.com> wrote:
+1 sounds good. Thanks Dongjoon for volunteering on this!
Liang-Chi
Dongjoon Hyun-2 wrote
> Hi, All.
>
> Since Apache Spark 3.1.1 tag creation (Feb 21),
> new 172 patches including 9 correctness patches and 4 K8s patches arrived
> at branch-3.1.
>
> Shall we make a new release, Apache Spark 3.1.2, as the second release at
> 3.1 line?
> I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> I'm thinking about starting the first RC next week.
>
> $ git log --oneline v3.1.1..HEAD | wc -l
> 172
>
> # Known correctness issues
> SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
> lead to data loss or correctness
> SPARK-34545 PySpark Python UDF return inconsistent results when
> applying 2 UDFs with different return type to 2 columns together
> SPARK-34681 Full outer shuffled hash join when building left side
> produces wrong result
> SPARK-34719 fail if the view query has duplicated column names
> SPARK-34794 Nested higher-order functions broken in DSL
> SPARK-34829 transform_values return identical values when it's used
> with udf that returns reference type
> SPARK-34833 Apply right-padding correctly for correlated subqueries
> SPARK-35381 Fix lambda variable name issues in nested DataFrame
> functions in R APIs
> SPARK-35382 Fix lambda variable name issues in nested DataFrame
> functions in Python APIs
>
> # Notable K8s patches since K8s GA
> SPARK-34674 Close SparkContext after the Main method has finished
> SPARK-34948 Add ownerReference to executor configmap to fix leakages
> SPARK-34820 add apt-update before gnupg install
> SPARK-34361 In case of downscaling avoid killing of executors already
> known by the scheduler backend in the pod allocator
>
> Bests,
> Dongjoon.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Apache Spark 3.1.2 Release?
Posted by Liang-Chi Hsieh <vi...@gmail.com>.
+1 sounds good. Thanks Dongjoon for volunteering on this!
Liang-Chi
Dongjoon Hyun-2 wrote
> Hi, All.
>
> Since Apache Spark 3.1.1 tag creation (Feb 21),
> new 172 patches including 9 correctness patches and 4 K8s patches arrived
> at branch-3.1.
>
> Shall we make a new release, Apache Spark 3.1.2, as the second release at
> 3.1 line?
> I'd like to volunteer for the release manager for Apache Spark 3.1.2.
> I'm thinking about starting the first RC next week.
>
> $ git log --oneline v3.1.1..HEAD | wc -l
> 172
>
> # Known correctness issues
> SPARK-34534 New protocol FetchShuffleBlocks in OneForOneBlockFetcher
> lead to data loss or correctness
> SPARK-34545 PySpark Python UDF return inconsistent results when
> applying 2 UDFs with different return type to 2 columns together
> SPARK-34681 Full outer shuffled hash join when building left side
> produces wrong result
> SPARK-34719 fail if the view query has duplicated column names
> SPARK-34794 Nested higher-order functions broken in DSL
> SPARK-34829 transform_values return identical values when it's used
> with udf that returns reference type
> SPARK-34833 Apply right-padding correctly for correlated subqueries
> SPARK-35381 Fix lambda variable name issues in nested DataFrame
> functions in R APIs
> SPARK-35382 Fix lambda variable name issues in nested DataFrame
> functions in Python APIs
>
> # Notable K8s patches since K8s GA
> SPARK-34674 Close SparkContext after the Main method has finished
> SPARK-34948 Add ownerReference to executor configmap to fix leakages
> SPARK-34820 add apt-update before gnupg install
> SPARK-34361 In case of downscaling avoid killing of executors already
> known by the scheduler backend in the pod allocator
>
> Bests,
> Dongjoon.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org