You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Marion <nm...@us.ibm.com> on 2020/09/01 13:12:18 UTC
RE: [VOTE] Release Spark 2.4.7 (RC1)
Hello,
I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there
is an active vote going through for it?
Regards,
NICHOLAS T. MARION
IBM Open Data Analytics for z/OS - CPO and Service Team Lead
Phone: 1-845-433-5010 | Tie-Line: 293-5010 IBM
E-mail: nmarion@us.ibm.com
Find me on: LinkedIn: 2455 South Rd
http://www.linkedin.com/in/nicholasmarion Poughkeepie, New York 12601-5400
United States
IBM Redbooks Silver Author
Data Science Foundations - Level 1
From: Tom Graves <tg...@yahoo.com.INVALID>
To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan
<cl...@gmail.com>
Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>,
Takeshi Yamamuro <li...@gmail.com>, Prashant Sharma
<sc...@gmail.com>
Date: 08/21/2020 03:09 PM
Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
There is a correctness issue with caching that should go into this if
possible: https://github.com/apache/spark/pull/29506
Tom
On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan
<cl...@gmail.com> wrote:
I think so. I don't see other bug reports for 2.4.
On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <nm...@us.ibm.com>
wrote:
It appears all 3 issues slated for Spark 2.4.7 have been merged. Should
we be looking at getting RC2 ready?
Regards,
NICHOLAS T. MARION
IBM Open Data Analytics for z/OS - CPO and Service Team Lead
Phone: 1-845-433-5010 | Tie-Line: 293-5010
E-mail: nmarion@us.ibm.com
Find me on: 2455 South Rd
Poughkeepie, New York
12601-5400
United States
Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI
REMOVEDXiao Li ---08/17/2020 11:33:30 AM---
https://issues.apache.org/jira/browse/SPARK-32609
From: Xiao Li <li...@databricks.com>
To: Prashant Sharma <sc...@gmail.com>
Cc: Takeshi Yamamuro <li...@gmail.com>, dev <de...@spark.apache.org>
Date: 08/17/2020 11:33 AM
Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to
fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
upcoming Spark 2.4.7 release.
Thanks,
Xiao
On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <sc...@gmail.com>
wrote:
Thanks for letting us know. So this vote is cancelled in favor of RC2.
On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <li...@gmail.com>
wrote:
Thanks for letting us know about the two issues above, Dongjoon.
----
I've checked the release materials (signatures, tag, ...) and it looks
fine, too.
Also, I run the tests on my local Mac (java 1.8.0) with the options
`-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
-Psparkr`
and they passed.
Bests,
Takeshi
On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <do...@gmail.com>
wrote:
Another instance is SPARK-31703 which filed on May 13th and the PR
arrived two days ago.
[SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
big endian platforms
https://github.com/apache/spark/pull/29383
It seems that the patch is already ready in this case.
I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark
2.4.7 and 3.0.1.
Bests,
Dongjoon.
On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca> wrote:
I'm going to go ahead and vote -0 then based on that then.
On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com>
wrote:
Hi, All.
Unfortunately, there is an on-going discussion about the new decimal
correctness.
Although we fixed one correctness issue at master and backported it
partially to 3.0/2.4, it turns out that it needs more patched to be
complete.
Please see https://github.com/apache/spark/pull/29125 for on-going
discussion for both 3.0/2.4.
[SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
overflowed value
I also confirmed that 2.4.7 RC1 is affected.
Bests,
Dongjoon.
On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.
On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
wrote:
>
> Please vote on releasing the following candidate as Apache Spark
version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1
PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND
"Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit
dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following
URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
--
---
Takeshi Yamamuro
--
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Spark 2.4.7 (RC1)
Posted by Micah Kornfield <em...@gmail.com>.
Thank you for the clarification.
On Wed, Sep 2, 2020 at 6:18 AM Sean Owen <sr...@gmail.com> wrote:
> If you mean, will there be another 2.4.8 release? yes eventually. I am
> sure we'll do maintenance releases into early next year.
> What is the policy for holding a release for more changes? generally for
> correctness issues and/or regressions vs the previous maintenance release.
> Those entail an element of judgment call; other really important changes
> might hold it up.
> That looks like a change that should go into 2.4.x but not
> necessarily block 2.4.7. It doesn't look ready to commit, even.
>
> On Wed, Sep 2, 2020 at 1:29 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Just curious what is the policy for releases off of the 2.4 branch?
>>
>> It would be nice if a resolution to SPARK-32708
>> <https://issues.apache.org/jira/browse/SPARK-32708> (Reusing exchanges
>> for DataSourceV2) [1] could make it into the release. But if it can't what
>> would determine the need for a new release?
>>
>> Thanks,
>> Micah
>>
>> [1] https://issues.apache.org/jira/browse/SPARK-32708
>>
>> On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <sc...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> FYI, currently there are no blockers to release 2.4.7 at the moment.
>>> Soon there will be RC3, I am awaiting some correctness bugs to be merged.
>>>
>>> Thanks,
>>>
>>> On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if
>>>> there is an active vote going through for it?
>>>>
>>>>
>>>> Regards,
>>>>
>>>> *NICHOLAS T. MARION *
>>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>>> ------------------------------
>>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>>> *Find me on:* [image: LinkedIn:
>>>> http://www.linkedin.com/in/nicholasmarion]
>>>> <http://www.linkedin.com/in/nicholasmarion>
>>>> [image: IBM]
>>>>
>>>> 2455 South Rd
>>>> Poughkeepie, New York 12601-5400
>>>> United States
>>>> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
>>>> Level 1]
>>>>
>>>>
>>>> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49
>>>> PM--- There is a correctness issue with caching that should go into]Tom
>>>> Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with
>>>> caching that should go into this if possible: https://urldefense.
>>>>
>>>> From: Tom Graves <tg...@yahoo.com.INVALID>
>>>> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <
>>>> cloud0fan@gmail.com>
>>>> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>,
>>>> Takeshi Yamamuro <li...@gmail.com>, Prashant Sharma <
>>>> scrapcodes@gmail.com>
>>>> Date: 08/21/2020 03:09 PM
>>>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>>>
>>>> ------------------------------
>>>>
>>>>
>>>>
>>>> There is a correctness issue with caching that should go into this if
>>>> possible: *https://github.com/apache/spark/pull/29506*
>>>> <https://github.com/apache/spark/pull/29506>
>>>>
>>>> Tom
>>>>
>>>> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
>>>> cloud0fan@gmail.com> wrote:
>>>>
>>>>
>>>> I think so. I don't see other bug reports for 2.4.
>>>>
>>>> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
>>>> <nm...@us.ibm.com>> wrote:
>>>>
>>>> It appears all 3 issues slated for Spark 2.4.7 have been merged.
>>>> Should we be looking at getting RC2 ready?
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> *NICHOLAS T. MARION *
>>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>>> ------------------------------
>>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>>> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>>>>
>>>>
>>>> 2455 South Rd
>>>> Poughkeepie, New York 12601-5400
>>>> United States
>>>>
>>>>
>>>>
>>>>
>>>> [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
>>>> AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
>>>> *https://issues.apache.org/jira/browse/SPARK-32609*
>>>> <https://issues.apache.org/jira/browse/SPARK-32609>
>>>>
>>>> From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
>>>> To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
>>>> Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com*
>>>> <li...@gmail.com>>, dev <*dev@spark.apache.org*
>>>> <de...@spark.apache.org>>
>>>> Date: 08/17/2020 11:33 AM
>>>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>>> ------------------------------
>>>>
>>>>
>>>>
>>>> *https://issues.apache.org/jira/browse/SPARK-32609*
>>>> <https://issues.apache.org/jira/browse/SPARK-32609> got merged.
>>>> This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in
>>>> the upcoming Spark 2.4.7 release.
>>>>
>>>> Thanks,
>>>>
>>>> Xiao
>>>>
>>>> On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
>>>> *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>>> Thanks for letting us know. So this vote is cancelled in favor of
>>>> RC2.
>>>>
>>>>
>>>>
>>>> On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>>>> *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>>>> Thanks for letting us know about the two issues above, Dongjoon.
>>>>
>>>> ----
>>>> I've checked the release materials (signatures, tag, ...) and it
>>>> looks fine, too.
>>>> Also, I run the tests on my local Mac (java 1.8.0) with the options
>>>> `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos
>>>> -Pkubernetes -Psparkr`
>>>> and they passed.
>>>>
>>>> Bests,
>>>> Takeshi
>>>>
>>>>
>>>>
>>>> On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>>>> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>>> Another instance is SPARK-31703 which filed on May 13th and the PR
>>>> arrived two days ago.
>>>>
>>>> [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>>>> big endian platforms
>>>> *https://github.com/apache/spark/pull/29383*
>>>> <https://github.com/apache/spark/pull/29383>
>>>>
>>>> It seems that the patch is already ready in this case.
>>>> I raised the priority of SPARK-31703 to `Blocker` for both Apache
>>>> Spark 2.4.7 and 3.0.1.
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>>
>>>> On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>>>> <ho...@pigscanfly.ca>> wrote:
>>>> I'm going to go ahead and vote -0 then based on that then.
>>>>
>>>> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>>>> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>>> Hi, All.
>>>>
>>>> Unfortunately, there is an on-going discussion about the new
>>>> decimal correctness.
>>>>
>>>> Although we fixed one correctness issue at master and backported it
>>>> partially to 3.0/2.4, it turns out that it needs more patched to be
>>>> complete.
>>>>
>>>> Please see *https://github.com/apache/spark/pull/29125*
>>>> <https://github.com/apache/spark/pull/29125> for on-going
>>>> discussion for both 3.0/2.4.
>>>>
>>>> [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>>> overflowed value
>>>>
>>>> I also confirmed that 2.4.7 RC1 is affected.
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>>
>>>> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>>>> <sr...@apache.org>> wrote:
>>>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>>> passes tests on a standard selection of profiles.
>>>>
>>>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
>>>> *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>>> >
>>>> > Please vote on releasing the following candidate as Apache Spark
>>>> version 2.4.7.
>>>> >
>>>> > The vote is open until Aug 9th at 9AM PST and passes if a
>>>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>> >
>>>> > [ ] +1 Release this package as Apache Spark 2.4.7
>>>> > [ ] -1 Do not release this package because ...
>>>> >
>>>> > To learn more about Apache Spark, please see
>>>> *http://spark.apache.org/* <http://spark.apache.org/>
>>>> >
>>>> > There are currently no issues targeting 2.4.7 (try project =
>>>> SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>>> Progress"))
>>>> >
>>>> > The tag to be voted on is v2.4.7-rc1 (commit
>>>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>>> > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>>>> <https://github.com/apache/spark/tree/v2.4.7-rc1>
>>>> >
>>>> > The release files, including signatures, digests, etc. can be
>>>> found at:
>>>> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>>>> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>>>> >
>>>> > Signatures used for Spark RCs can be found in this file:
>>>> > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>>>> <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>>>> >
>>>> > The staging repository for this release can be found at:
>>>> >
>>>> *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>>>> <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>>>> >
>>>> > The documentation corresponding to this release can be found at:
>>>> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>>>> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>>>> >
>>>> > The list of bug fixes going into 2.4.7 can be found at the
>>>> following URL:
>>>> > *https://s.apache.org/spark-v2.4.7-rc1*
>>>> <https://s.apache.org/spark-v2.4.7-rc1>
>>>> >
>>>> > This release is using the release script of the tag v2.4.7-rc1.
>>>> >
>>>> > FAQ
>>>> >
>>>> >
>>>> > =========================
>>>> > How can I help test this release?
>>>> > =========================
>>>> >
>>>> > If you are a Spark user, you can help us test this release by
>>>> taking
>>>> > an existing Spark workload and running on this release candidate,
>>>> then
>>>> > reporting any regressions.
>>>> >
>>>> > If you're working in PySpark you can set up a virtual env and
>>>> install
>>>> > the current RC and see if anything important breaks, in the
>>>> Java/Scala
>>>> > you can add the staging repository to your projects resolvers and
>>>> test
>>>> > with the RC (make sure to clean up the artifact cache
>>>> before/after so
>>>> > you don't end up building with an out of date RC going forward).
>>>> >
>>>> > ===========================================
>>>> > What should happen to JIRA tickets still targeting 2.4.7?
>>>> > ===========================================
>>>> >
>>>> > The current list of open tickets targeted at 2.4.7 can be found
>>>> at:
>>>> > *https://issues.apache.org/jira/projects/SPARK*
>>>> <https://issues.apache.org/jira/projects/SPARK> and search for
>>>> "Target Version/s" = 2.4.7
>>>> >
>>>> > Committers should look at those and triage. Extremely important
>>>> bug
>>>> > fixes, documentation, and API tweaks that impact compatibility
>>>> should
>>>> > be worked on immediately. Everything else please retarget to an
>>>> > appropriate release.
>>>> >
>>>> > ==================
>>>> > But my bug isn't fixed?
>>>> > ==================
>>>> >
>>>> > In order to make timely releases, we will typically not hold the
>>>> > release unless the bug in question is a regression from the
>>>> previous
>>>> > release. That being said, if there is something which is a
>>>> regression
>>>> > that has not been correctly targeted please ping me or a
>>>> committer to
>>>> > help target the issue.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>>>> <de...@spark.apache.org>
>>>>
>>>>
>>>> --
>>>> Twitter: *https://twitter.com/holdenkarau*
>>>> <https://twitter.com/holdenkarau>
>>>> Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
>>>> * <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>>>> <https://www.youtube.com/user/holdenkarau>
>>>>
>>>> --
>>>> ---
>>>> Takeshi Yamamuro
>>>>
>>>> --
>>>> <https://databricks.com/sparkaisummit/north-america>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>>
>>>>
Re: [VOTE] Release Spark 2.4.7 (RC1)
Posted by Sean Owen <sr...@gmail.com>.
If you mean, will there be another 2.4.8 release? yes eventually. I am sure
we'll do maintenance releases into early next year.
What is the policy for holding a release for more changes? generally for
correctness issues and/or regressions vs the previous maintenance release.
Those entail an element of judgment call; other really important changes
might hold it up.
That looks like a change that should go into 2.4.x but not
necessarily block 2.4.7. It doesn't look ready to commit, even.
On Wed, Sep 2, 2020 at 1:29 AM Micah Kornfield <em...@gmail.com>
wrote:
> Just curious what is the policy for releases off of the 2.4 branch?
>
> It would be nice if a resolution to SPARK-32708
> <https://issues.apache.org/jira/browse/SPARK-32708> (Reusing exchanges
> for DataSourceV2) [1] could make it into the release. But if it can't what
> would determine the need for a new release?
>
> Thanks,
> Micah
>
> [1] https://issues.apache.org/jira/browse/SPARK-32708
>
> On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <sc...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> FYI, currently there are no blockers to release 2.4.7 at the moment. Soon
>> there will be RC3, I am awaiting some correctness bugs to be merged.
>>
>> Thanks,
>>
>> On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if
>>> there is an active vote going through for it?
>>>
>>>
>>> Regards,
>>>
>>> *NICHOLAS T. MARION *
>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>> ------------------------------
>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>> *Find me on:* [image: LinkedIn:
>>> http://www.linkedin.com/in/nicholasmarion]
>>> <http://www.linkedin.com/in/nicholasmarion>
>>> [image: IBM]
>>>
>>> 2455 South Rd
>>> Poughkeepie, New York 12601-5400
>>> United States
>>> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
>>> Level 1]
>>>
>>>
>>> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49
>>> PM--- There is a correctness issue with caching that should go into]Tom
>>> Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with
>>> caching that should go into this if possible: https://urldefense.
>>>
>>> From: Tom Graves <tg...@yahoo.com.INVALID>
>>> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <
>>> cloud0fan@gmail.com>
>>> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>,
>>> Takeshi Yamamuro <li...@gmail.com>, Prashant Sharma <
>>> scrapcodes@gmail.com>
>>> Date: 08/21/2020 03:09 PM
>>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>>
>>> ------------------------------
>>>
>>>
>>>
>>> There is a correctness issue with caching that should go into this if
>>> possible: *https://github.com/apache/spark/pull/29506*
>>> <https://github.com/apache/spark/pull/29506>
>>>
>>> Tom
>>>
>>> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
>>> cloud0fan@gmail.com> wrote:
>>>
>>>
>>> I think so. I don't see other bug reports for 2.4.
>>>
>>> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
>>> <nm...@us.ibm.com>> wrote:
>>>
>>> It appears all 3 issues slated for Spark 2.4.7 have been merged.
>>> Should we be looking at getting RC2 ready?
>>>
>>>
>>>
>>> Regards,
>>>
>>> *NICHOLAS T. MARION *
>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>> ------------------------------
>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>>>
>>>
>>> 2455 South Rd
>>> Poughkeepie, New York 12601-5400
>>> United States
>>>
>>>
>>>
>>>
>>> [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
>>> AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
>>> *https://issues.apache.org/jira/browse/SPARK-32609*
>>> <https://issues.apache.org/jira/browse/SPARK-32609>
>>>
>>> From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
>>> To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
>>> Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com* <li...@gmail.com>>,
>>> dev <*dev@spark.apache.org* <de...@spark.apache.org>>
>>> Date: 08/17/2020 11:33 AM
>>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>> ------------------------------
>>>
>>>
>>>
>>> *https://issues.apache.org/jira/browse/SPARK-32609*
>>> <https://issues.apache.org/jira/browse/SPARK-32609> got merged. This
>>> is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
>>> upcoming Spark 2.4.7 release.
>>>
>>> Thanks,
>>>
>>> Xiao
>>>
>>> On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
>>> *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>> Thanks for letting us know. So this vote is cancelled in favor of
>>> RC2.
>>>
>>>
>>>
>>> On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>>> *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>>> Thanks for letting us know about the two issues above, Dongjoon.
>>>
>>> ----
>>> I've checked the release materials (signatures, tag, ...) and it
>>> looks fine, too.
>>> Also, I run the tests on my local Mac (java 1.8.0) with the options
>>> `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>>> -Psparkr`
>>> and they passed.
>>>
>>> Bests,
>>> Takeshi
>>>
>>>
>>>
>>> On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>>> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>> Another instance is SPARK-31703 which filed on May 13th and the PR
>>> arrived two days ago.
>>>
>>> [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>>> big endian platforms
>>> *https://github.com/apache/spark/pull/29383*
>>> <https://github.com/apache/spark/pull/29383>
>>>
>>> It seems that the patch is already ready in this case.
>>> I raised the priority of SPARK-31703 to `Blocker` for both Apache
>>> Spark 2.4.7 and 3.0.1.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>>> <ho...@pigscanfly.ca>> wrote:
>>> I'm going to go ahead and vote -0 then based on that then.
>>>
>>> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>>> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>> Hi, All.
>>>
>>> Unfortunately, there is an on-going discussion about the new decimal
>>> correctness.
>>>
>>> Although we fixed one correctness issue at master and backported it
>>> partially to 3.0/2.4, it turns out that it needs more patched to be
>>> complete.
>>>
>>> Please see *https://github.com/apache/spark/pull/29125*
>>> <https://github.com/apache/spark/pull/29125> for on-going discussion
>>> for both 3.0/2.4.
>>>
>>> [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>> overflowed value
>>>
>>> I also confirmed that 2.4.7 RC1 is affected.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>>> <sr...@apache.org>> wrote:
>>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>> passes tests on a standard selection of profiles.
>>>
>>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
>>> *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.7.
>>> >
>>> > The vote is open until Aug 9th at 9AM PST and passes if a majority
>>> +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 2.4.7
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see
>>> *http://spark.apache.org/* <http://spark.apache.org/>
>>> >
>>> > There are currently no issues targeting 2.4.7 (try project = SPARK
>>> AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>> Progress"))
>>> >
>>> > The tag to be voted on is v2.4.7-rc1 (commit
>>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>> > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>>> <https://github.com/apache/spark/tree/v2.4.7-rc1>
>>> >
>>> > The release files, including signatures, digests, etc. can be
>>> found at:
>>> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>>> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>>> <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>>> <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>>> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>>> >
>>> > The list of bug fixes going into 2.4.7 can be found at the
>>> following URL:
>>> > *https://s.apache.org/spark-v2.4.7-rc1*
>>> <https://s.apache.org/spark-v2.4.7-rc1>
>>> >
>>> > This release is using the release script of the tag v2.4.7-rc1.
>>> >
>>> > FAQ
>>> >
>>> >
>>> > =========================
>>> > How can I help test this release?
>>> > =========================
>>> >
>>> > If you are a Spark user, you can help us test this release by
>>> taking
>>> > an existing Spark workload and running on this release candidate,
>>> then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and
>>> install
>>> > the current RC and see if anything important breaks, in the
>>> Java/Scala
>>> > you can add the staging repository to your projects resolvers and
>>> test
>>> > with the RC (make sure to clean up the artifact cache before/after
>>> so
>>> > you don't end up building with an out of date RC going forward).
>>> >
>>> > ===========================================
>>> > What should happen to JIRA tickets still targeting 2.4.7?
>>> > ===========================================
>>> >
>>> > The current list of open tickets targeted at 2.4.7 can be found at:
>>> > *https://issues.apache.org/jira/projects/SPARK*
>>> <https://issues.apache.org/jira/projects/SPARK> and search for
>>> "Target Version/s" = 2.4.7
>>> >
>>> > Committers should look at those and triage. Extremely important bug
>>> > fixes, documentation, and API tweaks that impact compatibility
>>> should
>>> > be worked on immediately. Everything else please retarget to an
>>> > appropriate release.
>>> >
>>> > ==================
>>> > But my bug isn't fixed?
>>> > ==================
>>> >
>>> > In order to make timely releases, we will typically not hold the
>>> > release unless the bug in question is a regression from the
>>> previous
>>> > release. That being said, if there is something which is a
>>> regression
>>> > that has not been correctly targeted please ping me or a committer
>>> to
>>> > help target the issue.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>>> <de...@spark.apache.org>
>>>
>>>
>>> --
>>> Twitter: *https://twitter.com/holdenkarau*
>>> <https://twitter.com/holdenkarau>
>>> Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
>>> * <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>>> <https://www.youtube.com/user/holdenkarau>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>>> --
>>> <https://databricks.com/sparkaisummit/north-america>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>>
>>>
Re: [VOTE] Release Spark 2.4.7 (RC1)
Posted by Micah Kornfield <em...@gmail.com>.
Just curious what is the policy for releases off of the 2.4 branch?
It would be nice if a resolution to SPARK-32708
<https://issues.apache.org/jira/browse/SPARK-32708> (Reusing exchanges for
DataSourceV2) [1] could make it into the release. But if it can't what
would determine the need for a new release?
Thanks,
Micah
[1] https://issues.apache.org/jira/browse/SPARK-32708
On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <sc...@gmail.com>
wrote:
> Hi All,
>
> FYI, currently there are no blockers to release 2.4.7 at the moment. Soon
> there will be RC3, I am awaiting some correctness bugs to be merged.
>
> Thanks,
>
> On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com> wrote:
>
>> Hello,
>>
>> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if
>> there is an active vote going through for it?
>>
>>
>> Regards,
>>
>> *NICHOLAS T. MARION *
>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>> ------------------------------
>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>> *Find me on:* [image: LinkedIn:
>> http://www.linkedin.com/in/nicholasmarion]
>> <http://www.linkedin.com/in/nicholasmarion>
>> [image: IBM]
>>
>> 2455 South Rd
>> Poughkeepie, New York 12601-5400
>> United States
>> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
>> Level 1]
>>
>>
>> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM---
>> There is a correctness issue with caching that should go into]Tom Graves
>> ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that
>> should go into this if possible: https://urldefense.
>>
>> From: Tom Graves <tg...@yahoo.com.INVALID>
>> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <
>> cloud0fan@gmail.com>
>> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>, Takeshi
>> Yamamuro <li...@gmail.com>, Prashant Sharma <sc...@gmail.com>
>> Date: 08/21/2020 03:09 PM
>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>
>> ------------------------------
>>
>>
>>
>> There is a correctness issue with caching that should go into this if
>> possible: *https://github.com/apache/spark/pull/29506*
>> <https://github.com/apache/spark/pull/29506>
>>
>> Tom
>>
>> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
>> cloud0fan@gmail.com> wrote:
>>
>>
>> I think so. I don't see other bug reports for 2.4.
>>
>> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
>> <nm...@us.ibm.com>> wrote:
>>
>> It appears all 3 issues slated for Spark 2.4.7 have been merged.
>> Should we be looking at getting RC2 ready?
>>
>>
>>
>> Regards,
>>
>> *NICHOLAS T. MARION *
>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>> ------------------------------
>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>>
>>
>> 2455 South Rd
>> Poughkeepie, New York 12601-5400
>> United States
>>
>>
>>
>>
>> [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
>> AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
>> *https://issues.apache.org/jira/browse/SPARK-32609*
>> <https://issues.apache.org/jira/browse/SPARK-32609>
>>
>> From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
>> To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
>> Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com* <li...@gmail.com>>,
>> dev <*dev@spark.apache.org* <de...@spark.apache.org>>
>> Date: 08/17/2020 11:33 AM
>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>> ------------------------------
>>
>>
>>
>> *https://issues.apache.org/jira/browse/SPARK-32609*
>> <https://issues.apache.org/jira/browse/SPARK-32609> got merged. This
>> is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
>> upcoming Spark 2.4.7 release.
>>
>> Thanks,
>>
>> Xiao
>>
>> On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
>> *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>> Thanks for letting us know. So this vote is cancelled in favor of RC2.
>>
>>
>>
>> On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>> *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>> Thanks for letting us know about the two issues above, Dongjoon.
>>
>> ----
>> I've checked the release materials (signatures, tag, ...) and it
>> looks fine, too.
>> Also, I run the tests on my local Mac (java 1.8.0) with the options
>> `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>> -Psparkr`
>> and they passed.
>>
>> Bests,
>> Takeshi
>>
>>
>>
>> On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>> Another instance is SPARK-31703 which filed on May 13th and the PR
>> arrived two days ago.
>>
>> [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>> big endian platforms
>> *https://github.com/apache/spark/pull/29383*
>> <https://github.com/apache/spark/pull/29383>
>>
>> It seems that the patch is already ready in this case.
>> I raised the priority of SPARK-31703 to `Blocker` for both Apache
>> Spark 2.4.7 and 3.0.1.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>> <ho...@pigscanfly.ca>> wrote:
>> I'm going to go ahead and vote -0 then based on that then.
>>
>> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>> Hi, All.
>>
>> Unfortunately, there is an on-going discussion about the new decimal
>> correctness.
>>
>> Although we fixed one correctness issue at master and backported it
>> partially to 3.0/2.4, it turns out that it needs more patched to be
>> complete.
>>
>> Please see *https://github.com/apache/spark/pull/29125*
>> <https://github.com/apache/spark/pull/29125> for on-going discussion
>> for both 3.0/2.4.
>>
>> [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>> overflowed value
>>
>> I also confirmed that 2.4.7 RC1 is affected.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>> <sr...@apache.org>> wrote:
>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>> passes tests on a standard selection of profiles.
>>
>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <*scrapcodes@gmail.com*
>> <sc...@gmail.com>> wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.7.
>> >
>> > The vote is open until Aug 9th at 9AM PST and passes if a majority
>> +1 PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.4.7
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see
>> *http://spark.apache.org/* <http://spark.apache.org/>
>> >
>> > There are currently no issues targeting 2.4.7 (try project = SPARK
>> AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>> Progress"))
>> >
>> > The tag to be voted on is v2.4.7-rc1 (commit
>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>> > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>> <https://github.com/apache/spark/tree/v2.4.7-rc1>
>> >
>> > The release files, including signatures, digests, etc. can be found
>> at:
>> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>> <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>> >
>> > The staging repository for this release can be found at:
>> >
>> *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>> <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>> >
>> > The documentation corresponding to this release can be found at:
>> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>> >
>> > The list of bug fixes going into 2.4.7 can be found at the
>> following URL:
>> > *https://s.apache.org/spark-v2.4.7-rc1*
>> <https://s.apache.org/spark-v2.4.7-rc1>
>> >
>> > This release is using the release script of the tag v2.4.7-rc1.
>> >
>> > FAQ
>> >
>> >
>> > =========================
>> > How can I help test this release?
>> > =========================
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate,
>> then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and
>> install
>> > the current RC and see if anything important breaks, in the
>> Java/Scala
>> > you can add the staging repository to your projects resolvers and
>> test
>> > with the RC (make sure to clean up the artifact cache before/after
>> so
>> > you don't end up building with an out of date RC going forward).
>> >
>> > ===========================================
>> > What should happen to JIRA tickets still targeting 2.4.7?
>> > ===========================================
>> >
>> > The current list of open tickets targeted at 2.4.7 can be found at:
>> > *https://issues.apache.org/jira/projects/SPARK*
>> <https://issues.apache.org/jira/projects/SPARK> and search for
>> "Target Version/s" = 2.4.7
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility
>> should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==================
>> > But my bug isn't fixed?
>> > ==================
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a
>> regression
>> > that has not been correctly targeted please ping me or a committer
>> to
>> > help target the issue.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>> <de...@spark.apache.org>
>>
>>
>> --
>> Twitter: *https://twitter.com/holdenkarau*
>> <https://twitter.com/holdenkarau>
>> Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
>> * <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>> <https://www.youtube.com/user/holdenkarau>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>> --
>> <https://databricks.com/sparkaisummit/north-america>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>>
>>
Re: [VOTE] Release Spark 2.4.7 (RC1)
Posted by Prashant Sharma <sc...@gmail.com>.
Hi All,
FYI, currently there are no blockers to release 2.4.7 at the moment. Soon
there will be RC3, I am awaiting some correctness bugs to be merged.
Thanks,
On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com> wrote:
> Hello,
>
> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there
> is an active vote going through for it?
>
>
> Regards,
>
> *NICHOLAS T. MARION *
> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
> ------------------------------
> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
> *Find me on:* [image: LinkedIn: http://www.linkedin.com/in/nicholasmarion]
> <http://www.linkedin.com/in/nicholasmarion>
> [image: IBM]
>
> 2455 South Rd
> Poughkeepie, New York 12601-5400
> United States
> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
> Level 1]
>
>
> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM---
> There is a correctness issue with caching that should go into]Tom Graves
> ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that
> should go into this if possible: https://urldefense.
>
> From: Tom Graves <tg...@yahoo.com.INVALID>
> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <cloud0fan@gmail.com
> >
> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>, Takeshi
> Yamamuro <li...@gmail.com>, Prashant Sharma <sc...@gmail.com>
> Date: 08/21/2020 03:09 PM
> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>
> ------------------------------
>
>
>
> There is a correctness issue with caching that should go into this if
> possible: *https://github.com/apache/spark/pull/29506*
> <https://github.com/apache/spark/pull/29506>
>
> Tom
>
> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
> cloud0fan@gmail.com> wrote:
>
>
> I think so. I don't see other bug reports for 2.4.
>
> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
> <nm...@us.ibm.com>> wrote:
>
> It appears all 3 issues slated for Spark 2.4.7 have been merged.
> Should we be looking at getting RC2 ready?
>
>
>
> Regards,
>
> *NICHOLAS T. MARION *
> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
> ------------------------------
> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>
>
> 2455 South Rd
> Poughkeepie, New York 12601-5400
> United States
>
>
>
>
> [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
> AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
> *https://issues.apache.org/jira/browse/SPARK-32609*
> <https://issues.apache.org/jira/browse/SPARK-32609>
>
> From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
> To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
> Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com* <li...@gmail.com>>,
> dev <*dev@spark.apache.org* <de...@spark.apache.org>>
> Date: 08/17/2020 11:33 AM
> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
> ------------------------------
>
>
>
> *https://issues.apache.org/jira/browse/SPARK-32609*
> <https://issues.apache.org/jira/browse/SPARK-32609> got merged. This
> is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
> upcoming Spark 2.4.7 release.
>
> Thanks,
>
> Xiao
>
> On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <*scrapcodes@gmail.com*
> <sc...@gmail.com>> wrote:
> Thanks for letting us know. So this vote is cancelled in favor of RC2.
>
>
>
> On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
> *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
> Thanks for letting us know about the two issues above, Dongjoon.
>
> ----
> I've checked the release materials (signatures, tag, ...) and it looks
> fine, too.
> Also, I run the tests on my local Mac (java 1.8.0) with the options
> `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
> -Psparkr`
> and they passed.
>
> Bests,
> Takeshi
>
>
>
> On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
> Another instance is SPARK-31703 which filed on May 13th and the PR
> arrived two days ago.
>
> [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
> big endian platforms
> *https://github.com/apache/spark/pull/29383*
> <https://github.com/apache/spark/pull/29383>
>
> It seems that the patch is already ready in this case.
> I raised the priority of SPARK-31703 to `Blocker` for both Apache
> Spark 2.4.7 and 3.0.1.
>
> Bests,
> Dongjoon.
>
>
> On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
> <ho...@pigscanfly.ca>> wrote:
> I'm going to go ahead and vote -0 then based on that then.
>
> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
> *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
> Hi, All.
>
> Unfortunately, there is an on-going discussion about the new decimal
> correctness.
>
> Although we fixed one correctness issue at master and backported it
> partially to 3.0/2.4, it turns out that it needs more patched to be
> complete.
>
> Please see *https://github.com/apache/spark/pull/29125*
> <https://github.com/apache/spark/pull/29125> for on-going discussion
> for both 3.0/2.4.
>
> [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
> overflowed value
>
> I also confirmed that 2.4.7 RC1 is affected.
>
> Bests,
> Dongjoon.
>
>
> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
> <sr...@apache.org>> wrote:
> +1 from me. The same as usual. Licenses and sigs look OK, builds and
> passes tests on a standard selection of profiles.
>
> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <*scrapcodes@gmail.com*
> <sc...@gmail.com>> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark
> version 2.4.7.
> >
> > The vote is open until Aug 9th at 9AM PST and passes if a majority
> +1 PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.7
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> *http://spark.apache.org/* <http://spark.apache.org/>
> >
> > There are currently no issues targeting 2.4.7 (try project = SPARK
> AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
> Progress"))
> >
> > The tag to be voted on is v2.4.7-rc1 (commit
> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> > *https://github.com/apache/spark/tree/v2.4.7-rc1*
> <https://github.com/apache/spark/tree/v2.4.7-rc1>
> >
> > The release files, including signatures, digests, etc. can be found
> at:
> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
> >
> > Signatures used for Spark RCs can be found in this file:
> > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
> <https://dist.apache.org/repos/dist/dev/spark/KEYS>
> >
> > The staging repository for this release can be found at:
> >
> *https://repository.apache.org/content/repositories/orgapachespark-1352/*
> <https://repository.apache.org/content/repositories/orgapachespark-1352/>
> >
> > The documentation corresponding to this release can be found at:
> > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
> <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
> >
> > The list of bug fixes going into 2.4.7 can be found at the following
> URL:
> > *https://s.apache.org/spark-v2.4.7-rc1*
> <https://s.apache.org/spark-v2.4.7-rc1>
> >
> > This release is using the release script of the tag v2.4.7-rc1.
> >
> > FAQ
> >
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate,
> then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the
> Java/Scala
> > you can add the staging repository to your projects resolvers and
> test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with an out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.7?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.7 can be found at:
> > *https://issues.apache.org/jira/projects/SPARK*
> <https://issues.apache.org/jira/projects/SPARK> and search for "Target
> Version/s" = 2.4.7
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
> <de...@spark.apache.org>
>
>
> --
> Twitter: *https://twitter.com/holdenkarau*
> <https://twitter.com/holdenkarau>
> Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
> * <https://amzn.to/2MaRAG9>
> YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
> <https://www.youtube.com/user/holdenkarau>
>
> --
> ---
> Takeshi Yamamuro
>
> --
> <https://databricks.com/sparkaisummit/north-america>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
>