You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Prashant Sharma <sc...@gmail.com> on 2020/08/06 12:06:37 UTC

[VOTE] Release Spark 2.4.7 (RC1)

Please vote on releasing the following candidate as Apache Spark
version 2.4.7.

The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC
votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.7
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

There are currently no issues targeting 2.4.7 (try project = SPARK AND
"Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))

The tag to be voted on is v2.4.7-rc1 (commit
dc04bf53fe821b7a07f817966c6c173f3b3788c6):
https://github.com/apache/spark/tree/v2.4.7-rc1

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1352/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/

The list of bug fixes going into 2.4.7 can be found at the following URL:
https://s.apache.org/spark-v2.4.7-rc1

This release is using the release script of the tag v2.4.7-rc1.

FAQ


=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.7?
===========================================

The current list of open tickets targeted at 2.4.7 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 2.4.7

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Micah Kornfield <em...@gmail.com>.
Thank you for the clarification.

On Wed, Sep 2, 2020 at 6:18 AM Sean Owen <sr...@gmail.com> wrote:

> If you mean, will there be another 2.4.8 release? yes eventually. I am
> sure we'll do maintenance releases into early next year.
> What is the policy for holding a release for more changes? generally for
> correctness issues and/or regressions vs the previous maintenance release.
> Those entail an element of judgment call; other really important changes
> might hold it up.
> That looks like a change that should go into 2.4.x but not
> necessarily block 2.4.7. It doesn't look ready to commit, even.
>
> On Wed, Sep 2, 2020 at 1:29 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Just curious what is the policy for releases off of the 2.4 branch?
>>
>> It would be nice if a resolution to SPARK-32708
>> <https://issues.apache.org/jira/browse/SPARK-32708> (Reusing exchanges
>> for DataSourceV2) [1] could make it into the release.  But if it can't what
>> would determine the need for a new release?
>>
>> Thanks,
>> Micah
>>
>> [1]   https://issues.apache.org/jira/browse/SPARK-32708
>>
>> On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <sc...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> FYI, currently there are no blockers to release 2.4.7 at the moment.
>>> Soon there will be RC3, I am awaiting some correctness bugs to be merged.
>>>
>>> Thanks,
>>>
>>> On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if
>>>> there is an active vote going through for it?
>>>>
>>>>
>>>> Regards,
>>>>
>>>> *NICHOLAS T. MARION *
>>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>>> ------------------------------
>>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>>> *Find me on:* [image: LinkedIn:
>>>> http://www.linkedin.com/in/nicholasmarion]
>>>> <http://www.linkedin.com/in/nicholasmarion>
>>>> [image: IBM]
>>>>
>>>> 2455 South Rd
>>>> Poughkeepie, New York 12601-5400
>>>> United States
>>>> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
>>>> Level 1]
>>>>
>>>>
>>>> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49
>>>> PM--- There is a correctness issue with caching that should go into]Tom
>>>> Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with
>>>> caching that should go into this if possible: https://urldefense.
>>>>
>>>> From: Tom Graves <tg...@yahoo.com.INVALID>
>>>> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <
>>>> cloud0fan@gmail.com>
>>>> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>,
>>>> Takeshi Yamamuro <li...@gmail.com>, Prashant Sharma <
>>>> scrapcodes@gmail.com>
>>>> Date: 08/21/2020 03:09 PM
>>>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>>>
>>>> ------------------------------
>>>>
>>>>
>>>>
>>>> There is a correctness issue with caching that should go into this if
>>>> possible: *https://github.com/apache/spark/pull/29506*
>>>> <https://github.com/apache/spark/pull/29506>
>>>>
>>>> Tom
>>>>
>>>> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
>>>> cloud0fan@gmail.com> wrote:
>>>>
>>>>
>>>> I think so. I don't see other bug reports for 2.4.
>>>>
>>>> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
>>>> <nm...@us.ibm.com>> wrote:
>>>>
>>>>    It appears all 3 issues slated for Spark 2.4.7 have been merged.
>>>>    Should we be looking at getting RC2 ready?
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> *NICHOLAS T. MARION *
>>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>>> ------------------------------
>>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>>> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>>>>
>>>>
>>>> 2455 South Rd
>>>> Poughkeepie, New York 12601-5400
>>>> United States
>>>>
>>>>
>>>>
>>>>
>>>>    [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
>>>>    AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
>>>>    *https://issues.apache.org/jira/browse/SPARK-32609*
>>>>    <https://issues.apache.org/jira/browse/SPARK-32609>
>>>>
>>>>    From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
>>>>    To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
>>>>    Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com*
>>>>    <li...@gmail.com>>, dev <*dev@spark.apache.org*
>>>>    <de...@spark.apache.org>>
>>>>    Date: 08/17/2020 11:33 AM
>>>>    Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>>>    ------------------------------
>>>>
>>>>
>>>>
>>>> *https://issues.apache.org/jira/browse/SPARK-32609*
>>>>    <https://issues.apache.org/jira/browse/SPARK-32609> got merged.
>>>>    This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in
>>>>    the upcoming Spark 2.4.7 release.
>>>>
>>>>    Thanks,
>>>>
>>>>    Xiao
>>>>
>>>>    On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
>>>>    *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>>>    Thanks for letting us know. So this vote is cancelled in favor of
>>>>    RC2.
>>>>
>>>>
>>>>
>>>>    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>>>>    *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>>>>    Thanks for letting us know about the two issues above, Dongjoon.
>>>>
>>>>    ----
>>>>    I've checked the release materials (signatures, tag, ...) and it
>>>>    looks fine, too.
>>>>    Also, I run the tests on my local Mac (java 1.8.0) with the options
>>>>    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos
>>>>    -Pkubernetes -Psparkr`
>>>>    and they passed.
>>>>
>>>>    Bests,
>>>>    Takeshi
>>>>
>>>>
>>>>
>>>>    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>>>>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>>>    Another instance is SPARK-31703 which filed on May 13th and the PR
>>>>    arrived two days ago.
>>>>
>>>>    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>>>>    big endian platforms
>>>>    *https://github.com/apache/spark/pull/29383*
>>>>    <https://github.com/apache/spark/pull/29383>
>>>>
>>>>    It seems that the patch is already ready in this case.
>>>>    I raised the priority of SPARK-31703 to `Blocker` for both Apache
>>>>    Spark 2.4.7 and 3.0.1.
>>>>
>>>>    Bests,
>>>>    Dongjoon.
>>>>
>>>>
>>>>    On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>>>>    <ho...@pigscanfly.ca>> wrote:
>>>>    I'm going to go ahead and vote -0 then based on that then.
>>>>
>>>>    On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>>>>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>>>    Hi, All.
>>>>
>>>>    Unfortunately, there is an on-going discussion about the new
>>>>    decimal correctness.
>>>>
>>>>    Although we fixed one correctness issue at master and backported it
>>>>    partially to 3.0/2.4, it turns out that it needs more patched to be
>>>>    complete.
>>>>
>>>>    Please see *https://github.com/apache/spark/pull/29125*
>>>>    <https://github.com/apache/spark/pull/29125> for on-going
>>>>    discussion for both 3.0/2.4.
>>>>
>>>>    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>>>    overflowed value
>>>>
>>>>    I also confirmed that 2.4.7 RC1 is affected.
>>>>
>>>>    Bests,
>>>>    Dongjoon.
>>>>
>>>>
>>>>    On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>>>>    <sr...@apache.org>> wrote:
>>>>    +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>>>    passes tests on a standard selection of profiles.
>>>>
>>>>    On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
>>>>    *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>>>    >
>>>>    > Please vote on releasing the following candidate as Apache Spark
>>>>    version 2.4.7.
>>>>    >
>>>>    > The vote is open until Aug 9th at 9AM PST and passes if a
>>>>    majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>>    >
>>>>    > [ ] +1 Release this package as Apache Spark 2.4.7
>>>>    > [ ] -1 Do not release this package because ...
>>>>    >
>>>>    > To learn more about Apache Spark, please see
>>>>    *http://spark.apache.org/* <http://spark.apache.org/>
>>>>    >
>>>>    > There are currently no issues targeting 2.4.7 (try project =
>>>>    SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>>>    Progress"))
>>>>    >
>>>>    > The tag to be voted on is v2.4.7-rc1 (commit
>>>>    dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>>>    > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>>>>    <https://github.com/apache/spark/tree/v2.4.7-rc1>
>>>>    >
>>>>    > The release files, including signatures, digests, etc. can be
>>>>    found at:
>>>>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>>>>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>>>>    >
>>>>    > Signatures used for Spark RCs can be found in this file:
>>>>    > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>>>>    <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>>>>    >
>>>>    > The staging repository for this release can be found at:
>>>>    >
>>>>    *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>>>>    <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>>>>    >
>>>>    > The documentation corresponding to this release can be found at:
>>>>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>>>>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>>>>    >
>>>>    > The list of bug fixes going into 2.4.7 can be found at the
>>>>    following URL:
>>>>    > *https://s.apache.org/spark-v2.4.7-rc1*
>>>>    <https://s.apache.org/spark-v2.4.7-rc1>
>>>>    >
>>>>    > This release is using the release script of the tag v2.4.7-rc1.
>>>>    >
>>>>    > FAQ
>>>>    >
>>>>    >
>>>>    > =========================
>>>>    > How can I help test this release?
>>>>    > =========================
>>>>    >
>>>>    > If you are a Spark user, you can help us test this release by
>>>>    taking
>>>>    > an existing Spark workload and running on this release candidate,
>>>>    then
>>>>    > reporting any regressions.
>>>>    >
>>>>    > If you're working in PySpark you can set up a virtual env and
>>>>    install
>>>>    > the current RC and see if anything important breaks, in the
>>>>    Java/Scala
>>>>    > you can add the staging repository to your projects resolvers and
>>>>    test
>>>>    > with the RC (make sure to clean up the artifact cache
>>>>    before/after so
>>>>    > you don't end up building with an out of date RC going forward).
>>>>    >
>>>>    > ===========================================
>>>>    > What should happen to JIRA tickets still targeting 2.4.7?
>>>>    > ===========================================
>>>>    >
>>>>    > The current list of open tickets targeted at 2.4.7 can be found
>>>>    at:
>>>>    > *https://issues.apache.org/jira/projects/SPARK*
>>>>    <https://issues.apache.org/jira/projects/SPARK> and search for
>>>>    "Target Version/s" = 2.4.7
>>>>    >
>>>>    > Committers should look at those and triage. Extremely important
>>>>    bug
>>>>    > fixes, documentation, and API tweaks that impact compatibility
>>>>    should
>>>>    > be worked on immediately. Everything else please retarget to an
>>>>    > appropriate release.
>>>>    >
>>>>    > ==================
>>>>    > But my bug isn't fixed?
>>>>    > ==================
>>>>    >
>>>>    > In order to make timely releases, we will typically not hold the
>>>>    > release unless the bug in question is a regression from the
>>>>    previous
>>>>    > release. That being said, if there is something which is a
>>>>    regression
>>>>    > that has not been correctly targeted please ping me or a
>>>>    committer to
>>>>    > help target the issue.
>>>>
>>>>
>>>>    ---------------------------------------------------------------------
>>>>    To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>>>>    <de...@spark.apache.org>
>>>>
>>>>
>>>>    --
>>>>    Twitter: *https://twitter.com/holdenkarau*
>>>>    <https://twitter.com/holdenkarau>
>>>>    Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
>>>>    * <https://amzn.to/2MaRAG9>
>>>>    YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>>>>    <https://www.youtube.com/user/holdenkarau>
>>>>
>>>>    --
>>>>    ---
>>>>    Takeshi Yamamuro
>>>>
>>>>    --
>>>> <https://databricks.com/sparkaisummit/north-america>
>>>>
>>>>
>>>>
>>>>    ---------------------------------------------------------------------
>>>>    To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>>
>>>>

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Sean Owen <sr...@gmail.com>.
If you mean, will there be another 2.4.8 release? yes eventually. I am sure
we'll do maintenance releases into early next year.
What is the policy for holding a release for more changes? generally for
correctness issues and/or regressions vs the previous maintenance release.
Those entail an element of judgment call; other really important changes
might hold it up.
That looks like a change that should go into 2.4.x but not
necessarily block 2.4.7. It doesn't look ready to commit, even.

On Wed, Sep 2, 2020 at 1:29 AM Micah Kornfield <em...@gmail.com>
wrote:

> Just curious what is the policy for releases off of the 2.4 branch?
>
> It would be nice if a resolution to SPARK-32708
> <https://issues.apache.org/jira/browse/SPARK-32708> (Reusing exchanges
> for DataSourceV2) [1] could make it into the release.  But if it can't what
> would determine the need for a new release?
>
> Thanks,
> Micah
>
> [1]   https://issues.apache.org/jira/browse/SPARK-32708
>
> On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <sc...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> FYI, currently there are no blockers to release 2.4.7 at the moment. Soon
>> there will be RC3, I am awaiting some correctness bugs to be merged.
>>
>> Thanks,
>>
>> On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if
>>> there is an active vote going through for it?
>>>
>>>
>>> Regards,
>>>
>>> *NICHOLAS T. MARION *
>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>> ------------------------------
>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>> *Find me on:* [image: LinkedIn:
>>> http://www.linkedin.com/in/nicholasmarion]
>>> <http://www.linkedin.com/in/nicholasmarion>
>>> [image: IBM]
>>>
>>> 2455 South Rd
>>> Poughkeepie, New York 12601-5400
>>> United States
>>> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
>>> Level 1]
>>>
>>>
>>> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49
>>> PM--- There is a correctness issue with caching that should go into]Tom
>>> Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with
>>> caching that should go into this if possible: https://urldefense.
>>>
>>> From: Tom Graves <tg...@yahoo.com.INVALID>
>>> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <
>>> cloud0fan@gmail.com>
>>> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>,
>>> Takeshi Yamamuro <li...@gmail.com>, Prashant Sharma <
>>> scrapcodes@gmail.com>
>>> Date: 08/21/2020 03:09 PM
>>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>>
>>> ------------------------------
>>>
>>>
>>>
>>> There is a correctness issue with caching that should go into this if
>>> possible: *https://github.com/apache/spark/pull/29506*
>>> <https://github.com/apache/spark/pull/29506>
>>>
>>> Tom
>>>
>>> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
>>> cloud0fan@gmail.com> wrote:
>>>
>>>
>>> I think so. I don't see other bug reports for 2.4.
>>>
>>> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
>>> <nm...@us.ibm.com>> wrote:
>>>
>>>    It appears all 3 issues slated for Spark 2.4.7 have been merged.
>>>    Should we be looking at getting RC2 ready?
>>>
>>>
>>>
>>> Regards,
>>>
>>> *NICHOLAS T. MARION *
>>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>>> ------------------------------
>>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>>> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>>>
>>>
>>> 2455 South Rd
>>> Poughkeepie, New York 12601-5400
>>> United States
>>>
>>>
>>>
>>>
>>>    [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
>>>    AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
>>>    *https://issues.apache.org/jira/browse/SPARK-32609*
>>>    <https://issues.apache.org/jira/browse/SPARK-32609>
>>>
>>>    From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
>>>    To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
>>>    Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com* <li...@gmail.com>>,
>>>    dev <*dev@spark.apache.org* <de...@spark.apache.org>>
>>>    Date: 08/17/2020 11:33 AM
>>>    Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>>    ------------------------------
>>>
>>>
>>>
>>> *https://issues.apache.org/jira/browse/SPARK-32609*
>>>    <https://issues.apache.org/jira/browse/SPARK-32609> got merged. This
>>>    is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
>>>    upcoming Spark 2.4.7 release.
>>>
>>>    Thanks,
>>>
>>>    Xiao
>>>
>>>    On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
>>>    *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>>    Thanks for letting us know. So this vote is cancelled in favor of
>>>    RC2.
>>>
>>>
>>>
>>>    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>>>    *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>>>    Thanks for letting us know about the two issues above, Dongjoon.
>>>
>>>    ----
>>>    I've checked the release materials (signatures, tag, ...) and it
>>>    looks fine, too.
>>>    Also, I run the tests on my local Mac (java 1.8.0) with the options
>>>    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>>>    -Psparkr`
>>>    and they passed.
>>>
>>>    Bests,
>>>    Takeshi
>>>
>>>
>>>
>>>    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>>>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>>    Another instance is SPARK-31703 which filed on May 13th and the PR
>>>    arrived two days ago.
>>>
>>>    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>>>    big endian platforms
>>>    *https://github.com/apache/spark/pull/29383*
>>>    <https://github.com/apache/spark/pull/29383>
>>>
>>>    It seems that the patch is already ready in this case.
>>>    I raised the priority of SPARK-31703 to `Blocker` for both Apache
>>>    Spark 2.4.7 and 3.0.1.
>>>
>>>    Bests,
>>>    Dongjoon.
>>>
>>>
>>>    On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>>>    <ho...@pigscanfly.ca>> wrote:
>>>    I'm going to go ahead and vote -0 then based on that then.
>>>
>>>    On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>>>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>>    Hi, All.
>>>
>>>    Unfortunately, there is an on-going discussion about the new decimal
>>>    correctness.
>>>
>>>    Although we fixed one correctness issue at master and backported it
>>>    partially to 3.0/2.4, it turns out that it needs more patched to be
>>>    complete.
>>>
>>>    Please see *https://github.com/apache/spark/pull/29125*
>>>    <https://github.com/apache/spark/pull/29125> for on-going discussion
>>>    for both 3.0/2.4.
>>>
>>>    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>>    overflowed value
>>>
>>>    I also confirmed that 2.4.7 RC1 is affected.
>>>
>>>    Bests,
>>>    Dongjoon.
>>>
>>>
>>>    On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>>>    <sr...@apache.org>> wrote:
>>>    +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>>    passes tests on a standard selection of profiles.
>>>
>>>    On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
>>>    *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>>    >
>>>    > Please vote on releasing the following candidate as Apache Spark
>>>    version 2.4.7.
>>>    >
>>>    > The vote is open until Aug 9th at 9AM PST and passes if a majority
>>>    +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>    >
>>>    > [ ] +1 Release this package as Apache Spark 2.4.7
>>>    > [ ] -1 Do not release this package because ...
>>>    >
>>>    > To learn more about Apache Spark, please see
>>>    *http://spark.apache.org/* <http://spark.apache.org/>
>>>    >
>>>    > There are currently no issues targeting 2.4.7 (try project = SPARK
>>>    AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>>    Progress"))
>>>    >
>>>    > The tag to be voted on is v2.4.7-rc1 (commit
>>>    dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>>    > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>>>    <https://github.com/apache/spark/tree/v2.4.7-rc1>
>>>    >
>>>    > The release files, including signatures, digests, etc. can be
>>>    found at:
>>>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>>>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>>>    >
>>>    > Signatures used for Spark RCs can be found in this file:
>>>    > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>>>    <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>>>    >
>>>    > The staging repository for this release can be found at:
>>>    >
>>>    *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>>>    <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>>>    >
>>>    > The documentation corresponding to this release can be found at:
>>>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>>>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>>>    >
>>>    > The list of bug fixes going into 2.4.7 can be found at the
>>>    following URL:
>>>    > *https://s.apache.org/spark-v2.4.7-rc1*
>>>    <https://s.apache.org/spark-v2.4.7-rc1>
>>>    >
>>>    > This release is using the release script of the tag v2.4.7-rc1.
>>>    >
>>>    > FAQ
>>>    >
>>>    >
>>>    > =========================
>>>    > How can I help test this release?
>>>    > =========================
>>>    >
>>>    > If you are a Spark user, you can help us test this release by
>>>    taking
>>>    > an existing Spark workload and running on this release candidate,
>>>    then
>>>    > reporting any regressions.
>>>    >
>>>    > If you're working in PySpark you can set up a virtual env and
>>>    install
>>>    > the current RC and see if anything important breaks, in the
>>>    Java/Scala
>>>    > you can add the staging repository to your projects resolvers and
>>>    test
>>>    > with the RC (make sure to clean up the artifact cache before/after
>>>    so
>>>    > you don't end up building with an out of date RC going forward).
>>>    >
>>>    > ===========================================
>>>    > What should happen to JIRA tickets still targeting 2.4.7?
>>>    > ===========================================
>>>    >
>>>    > The current list of open tickets targeted at 2.4.7 can be found at:
>>>    > *https://issues.apache.org/jira/projects/SPARK*
>>>    <https://issues.apache.org/jira/projects/SPARK> and search for
>>>    "Target Version/s" = 2.4.7
>>>    >
>>>    > Committers should look at those and triage. Extremely important bug
>>>    > fixes, documentation, and API tweaks that impact compatibility
>>>    should
>>>    > be worked on immediately. Everything else please retarget to an
>>>    > appropriate release.
>>>    >
>>>    > ==================
>>>    > But my bug isn't fixed?
>>>    > ==================
>>>    >
>>>    > In order to make timely releases, we will typically not hold the
>>>    > release unless the bug in question is a regression from the
>>>    previous
>>>    > release. That being said, if there is something which is a
>>>    regression
>>>    > that has not been correctly targeted please ping me or a committer
>>>    to
>>>    > help target the issue.
>>>
>>>    ---------------------------------------------------------------------
>>>    To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>>>    <de...@spark.apache.org>
>>>
>>>
>>>    --
>>>    Twitter: *https://twitter.com/holdenkarau*
>>>    <https://twitter.com/holdenkarau>
>>>    Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
>>>    * <https://amzn.to/2MaRAG9>
>>>    YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>>>    <https://www.youtube.com/user/holdenkarau>
>>>
>>>    --
>>>    ---
>>>    Takeshi Yamamuro
>>>
>>>    --
>>> <https://databricks.com/sparkaisummit/north-america>
>>>
>>>
>>>    ---------------------------------------------------------------------
>>>    To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>>
>>>

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Micah Kornfield <em...@gmail.com>.
Just curious what is the policy for releases off of the 2.4 branch?

It would be nice if a resolution to SPARK-32708
<https://issues.apache.org/jira/browse/SPARK-32708> (Reusing exchanges for
DataSourceV2) [1] could make it into the release.  But if it can't what
would determine the need for a new release?

Thanks,
Micah

[1]   https://issues.apache.org/jira/browse/SPARK-32708

On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <sc...@gmail.com>
wrote:

> Hi All,
>
> FYI, currently there are no blockers to release 2.4.7 at the moment. Soon
> there will be RC3, I am awaiting some correctness bugs to be merged.
>
> Thanks,
>
> On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com> wrote:
>
>> Hello,
>>
>> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if
>> there is an active vote going through for it?
>>
>>
>> Regards,
>>
>> *NICHOLAS T. MARION *
>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>> ------------------------------
>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>> *Find me on:* [image: LinkedIn:
>> http://www.linkedin.com/in/nicholasmarion]
>> <http://www.linkedin.com/in/nicholasmarion>
>> [image: IBM]
>>
>> 2455 South Rd
>> Poughkeepie, New York 12601-5400
>> United States
>> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
>> Level 1]
>>
>>
>> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM---
>> There is a correctness issue with caching that should go into]Tom Graves
>> ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that
>> should go into this if possible: https://urldefense.
>>
>> From: Tom Graves <tg...@yahoo.com.INVALID>
>> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <
>> cloud0fan@gmail.com>
>> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>, Takeshi
>> Yamamuro <li...@gmail.com>, Prashant Sharma <sc...@gmail.com>
>> Date: 08/21/2020 03:09 PM
>> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>
>> ------------------------------
>>
>>
>>
>> There is a correctness issue with caching that should go into this if
>> possible: *https://github.com/apache/spark/pull/29506*
>> <https://github.com/apache/spark/pull/29506>
>>
>> Tom
>>
>> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
>> cloud0fan@gmail.com> wrote:
>>
>>
>> I think so. I don't see other bug reports for 2.4.
>>
>> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
>> <nm...@us.ibm.com>> wrote:
>>
>>    It appears all 3 issues slated for Spark 2.4.7 have been merged.
>>    Should we be looking at getting RC2 ready?
>>
>>
>>
>> Regards,
>>
>> *NICHOLAS T. MARION *
>> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
>> ------------------------------
>> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
>> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
>> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>>
>>
>> 2455 South Rd
>> Poughkeepie, New York 12601-5400
>> United States
>>
>>
>>
>>
>>    [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
>>    AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
>>    *https://issues.apache.org/jira/browse/SPARK-32609*
>>    <https://issues.apache.org/jira/browse/SPARK-32609>
>>
>>    From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
>>    To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
>>    Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com* <li...@gmail.com>>,
>>    dev <*dev@spark.apache.org* <de...@spark.apache.org>>
>>    Date: 08/17/2020 11:33 AM
>>    Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>>    ------------------------------
>>
>>
>>
>> *https://issues.apache.org/jira/browse/SPARK-32609*
>>    <https://issues.apache.org/jira/browse/SPARK-32609> got merged. This
>>    is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
>>    upcoming Spark 2.4.7 release.
>>
>>    Thanks,
>>
>>    Xiao
>>
>>    On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
>>    *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>>    Thanks for letting us know. So this vote is cancelled in favor of RC2.
>>
>>
>>
>>    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>>    *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>>    Thanks for letting us know about the two issues above, Dongjoon.
>>
>>    ----
>>    I've checked the release materials (signatures, tag, ...) and it
>>    looks fine, too.
>>    Also, I run the tests on my local Mac (java 1.8.0) with the options
>>    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>>    -Psparkr`
>>    and they passed.
>>
>>    Bests,
>>    Takeshi
>>
>>
>>
>>    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>    Another instance is SPARK-31703 which filed on May 13th and the PR
>>    arrived two days ago.
>>
>>    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>>    big endian platforms
>>    *https://github.com/apache/spark/pull/29383*
>>    <https://github.com/apache/spark/pull/29383>
>>
>>    It seems that the patch is already ready in this case.
>>    I raised the priority of SPARK-31703 to `Blocker` for both Apache
>>    Spark 2.4.7 and 3.0.1.
>>
>>    Bests,
>>    Dongjoon.
>>
>>
>>    On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>>    <ho...@pigscanfly.ca>> wrote:
>>    I'm going to go ahead and vote -0 then based on that then.
>>
>>    On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>>    Hi, All.
>>
>>    Unfortunately, there is an on-going discussion about the new decimal
>>    correctness.
>>
>>    Although we fixed one correctness issue at master and backported it
>>    partially to 3.0/2.4, it turns out that it needs more patched to be
>>    complete.
>>
>>    Please see *https://github.com/apache/spark/pull/29125*
>>    <https://github.com/apache/spark/pull/29125> for on-going discussion
>>    for both 3.0/2.4.
>>
>>    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>    overflowed value
>>
>>    I also confirmed that 2.4.7 RC1 is affected.
>>
>>    Bests,
>>    Dongjoon.
>>
>>
>>    On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>>    <sr...@apache.org>> wrote:
>>    +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>    passes tests on a standard selection of profiles.
>>
>>    On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <*scrapcodes@gmail.com*
>>    <sc...@gmail.com>> wrote:
>>    >
>>    > Please vote on releasing the following candidate as Apache Spark
>>    version 2.4.7.
>>    >
>>    > The vote is open until Aug 9th at 9AM PST and passes if a majority
>>    +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>    >
>>    > [ ] +1 Release this package as Apache Spark 2.4.7
>>    > [ ] -1 Do not release this package because ...
>>    >
>>    > To learn more about Apache Spark, please see
>>    *http://spark.apache.org/* <http://spark.apache.org/>
>>    >
>>    > There are currently no issues targeting 2.4.7 (try project = SPARK
>>    AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>    Progress"))
>>    >
>>    > The tag to be voted on is v2.4.7-rc1 (commit
>>    dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>    > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>>    <https://github.com/apache/spark/tree/v2.4.7-rc1>
>>    >
>>    > The release files, including signatures, digests, etc. can be found
>>    at:
>>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>>    >
>>    > Signatures used for Spark RCs can be found in this file:
>>    > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>>    <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>>    >
>>    > The staging repository for this release can be found at:
>>    >
>>    *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>>    <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>>    >
>>    > The documentation corresponding to this release can be found at:
>>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>>    >
>>    > The list of bug fixes going into 2.4.7 can be found at the
>>    following URL:
>>    > *https://s.apache.org/spark-v2.4.7-rc1*
>>    <https://s.apache.org/spark-v2.4.7-rc1>
>>    >
>>    > This release is using the release script of the tag v2.4.7-rc1.
>>    >
>>    > FAQ
>>    >
>>    >
>>    > =========================
>>    > How can I help test this release?
>>    > =========================
>>    >
>>    > If you are a Spark user, you can help us test this release by taking
>>    > an existing Spark workload and running on this release candidate,
>>    then
>>    > reporting any regressions.
>>    >
>>    > If you're working in PySpark you can set up a virtual env and
>>    install
>>    > the current RC and see if anything important breaks, in the
>>    Java/Scala
>>    > you can add the staging repository to your projects resolvers and
>>    test
>>    > with the RC (make sure to clean up the artifact cache before/after
>>    so
>>    > you don't end up building with an out of date RC going forward).
>>    >
>>    > ===========================================
>>    > What should happen to JIRA tickets still targeting 2.4.7?
>>    > ===========================================
>>    >
>>    > The current list of open tickets targeted at 2.4.7 can be found at:
>>    > *https://issues.apache.org/jira/projects/SPARK*
>>    <https://issues.apache.org/jira/projects/SPARK> and search for
>>    "Target Version/s" = 2.4.7
>>    >
>>    > Committers should look at those and triage. Extremely important bug
>>    > fixes, documentation, and API tweaks that impact compatibility
>>    should
>>    > be worked on immediately. Everything else please retarget to an
>>    > appropriate release.
>>    >
>>    > ==================
>>    > But my bug isn't fixed?
>>    > ==================
>>    >
>>    > In order to make timely releases, we will typically not hold the
>>    > release unless the bug in question is a regression from the previous
>>    > release. That being said, if there is something which is a
>>    regression
>>    > that has not been correctly targeted please ping me or a committer
>>    to
>>    > help target the issue.
>>
>>    ---------------------------------------------------------------------
>>    To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>>    <de...@spark.apache.org>
>>
>>
>>    --
>>    Twitter: *https://twitter.com/holdenkarau*
>>    <https://twitter.com/holdenkarau>
>>    Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
>>    * <https://amzn.to/2MaRAG9>
>>    YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>>    <https://www.youtube.com/user/holdenkarau>
>>
>>    --
>>    ---
>>    Takeshi Yamamuro
>>
>>    --
>> <https://databricks.com/sparkaisummit/north-america>
>>
>>
>>    ---------------------------------------------------------------------
>>    To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>>
>>

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Prashant Sharma <sc...@gmail.com>.
Hi All,

FYI, currently there are no blockers to release 2.4.7 at the moment. Soon
there will be RC3, I am awaiting some correctness bugs to be merged.

Thanks,

On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <nm...@us.ibm.com> wrote:

> Hello,
>
> I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there
> is an active vote going through for it?
>
>
> Regards,
>
> *NICHOLAS T. MARION *
> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
> ------------------------------
> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
> *Find me on:* [image: LinkedIn: http://www.linkedin.com/in/nicholasmarion]
> <http://www.linkedin.com/in/nicholasmarion>
> [image: IBM]
>
> 2455 South Rd
> Poughkeepie, New York 12601-5400
> United States
> [image: IBM Redbooks Silver Author][image: Data Science Foundations -
> Level 1]
>
>
> [image: Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM---
> There is a correctness issue with caching that should go into]Tom Graves
> ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that
> should go into this if possible: https://urldefense.
>
> From: Tom Graves <tg...@yahoo.com.INVALID>
> To: Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan <cloud0fan@gmail.com
> >
> Cc: Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>, Takeshi
> Yamamuro <li...@gmail.com>, Prashant Sharma <sc...@gmail.com>
> Date: 08/21/2020 03:09 PM
> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>
> ------------------------------
>
>
>
> There is a correctness issue with caching that should go into this if
> possible: *https://github.com/apache/spark/pull/29506*
> <https://github.com/apache/spark/pull/29506>
>
> Tom
>
> On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <
> cloud0fan@gmail.com> wrote:
>
>
> I think so. I don't see other bug reports for 2.4.
>
> On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <*nmarion@us.ibm.com*
> <nm...@us.ibm.com>> wrote:
>
>    It appears all 3 issues slated for Spark 2.4.7 have been merged.
>    Should we be looking at getting RC2 ready?
>
>
>
> Regards,
>
> *NICHOLAS T. MARION *
> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
> ------------------------------
> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>
>
> 2455 South Rd
> Poughkeepie, New York 12601-5400
> United States
>
>
>
>
>    [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
>    AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
>    *https://issues.apache.org/jira/browse/SPARK-32609*
>    <https://issues.apache.org/jira/browse/SPARK-32609>
>
>    From: Xiao Li <*lixiao@databricks.com* <li...@databricks.com>>
>    To: Prashant Sharma <*scrapcodes@gmail.com* <sc...@gmail.com>>
>    Cc: Takeshi Yamamuro <*linguin.m.s@gmail.com* <li...@gmail.com>>,
>    dev <*dev@spark.apache.org* <de...@spark.apache.org>>
>    Date: 08/17/2020 11:33 AM
>    Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>    ------------------------------
>
>
>
> *https://issues.apache.org/jira/browse/SPARK-32609*
>    <https://issues.apache.org/jira/browse/SPARK-32609> got merged. This
>    is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
>    upcoming Spark 2.4.7 release.
>
>    Thanks,
>
>    Xiao
>
>    On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <*scrapcodes@gmail.com*
>    <sc...@gmail.com>> wrote:
>    Thanks for letting us know. So this vote is cancelled in favor of RC2.
>
>
>
>    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>    *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>    Thanks for letting us know about the two issues above, Dongjoon.
>
>    ----
>    I've checked the release materials (signatures, tag, ...) and it looks
>    fine, too.
>    Also, I run the tests on my local Mac (java 1.8.0) with the options
>    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>    -Psparkr`
>    and they passed.
>
>    Bests,
>    Takeshi
>
>
>
>    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>    Another instance is SPARK-31703 which filed on May 13th and the PR
>    arrived two days ago.
>
>    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>    big endian platforms
>    *https://github.com/apache/spark/pull/29383*
>    <https://github.com/apache/spark/pull/29383>
>
>    It seems that the patch is already ready in this case.
>    I raised the priority of SPARK-31703 to `Blocker` for both Apache
>    Spark 2.4.7 and 3.0.1.
>
>    Bests,
>    Dongjoon.
>
>
>    On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>    <ho...@pigscanfly.ca>> wrote:
>    I'm going to go ahead and vote -0 then based on that then.
>
>    On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>    Hi, All.
>
>    Unfortunately, there is an on-going discussion about the new decimal
>    correctness.
>
>    Although we fixed one correctness issue at master and backported it
>    partially to 3.0/2.4, it turns out that it needs more patched to be
>    complete.
>
>    Please see *https://github.com/apache/spark/pull/29125*
>    <https://github.com/apache/spark/pull/29125> for on-going discussion
>    for both 3.0/2.4.
>
>    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>    overflowed value
>
>    I also confirmed that 2.4.7 RC1 is affected.
>
>    Bests,
>    Dongjoon.
>
>
>    On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>    <sr...@apache.org>> wrote:
>    +1 from me. The same as usual. Licenses and sigs look OK, builds and
>    passes tests on a standard selection of profiles.
>
>    On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <*scrapcodes@gmail.com*
>    <sc...@gmail.com>> wrote:
>    >
>    > Please vote on releasing the following candidate as Apache Spark
>    version 2.4.7.
>    >
>    > The vote is open until Aug 9th at 9AM PST and passes if a majority
>    +1 PMC votes are cast, with a minimum of 3 +1 votes.
>    >
>    > [ ] +1 Release this package as Apache Spark 2.4.7
>    > [ ] -1 Do not release this package because ...
>    >
>    > To learn more about Apache Spark, please see
>    *http://spark.apache.org/* <http://spark.apache.org/>
>    >
>    > There are currently no issues targeting 2.4.7 (try project = SPARK
>    AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>    Progress"))
>    >
>    > The tag to be voted on is v2.4.7-rc1 (commit
>    dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>    > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>    <https://github.com/apache/spark/tree/v2.4.7-rc1>
>    >
>    > The release files, including signatures, digests, etc. can be found
>    at:
>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>    >
>    > Signatures used for Spark RCs can be found in this file:
>    > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>    <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>    >
>    > The staging repository for this release can be found at:
>    >
>    *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>    <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>    >
>    > The documentation corresponding to this release can be found at:
>    > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>    <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>    >
>    > The list of bug fixes going into 2.4.7 can be found at the following
>    URL:
>    > *https://s.apache.org/spark-v2.4.7-rc1*
>    <https://s.apache.org/spark-v2.4.7-rc1>
>    >
>    > This release is using the release script of the tag v2.4.7-rc1.
>    >
>    > FAQ
>    >
>    >
>    > =========================
>    > How can I help test this release?
>    > =========================
>    >
>    > If you are a Spark user, you can help us test this release by taking
>    > an existing Spark workload and running on this release candidate,
>    then
>    > reporting any regressions.
>    >
>    > If you're working in PySpark you can set up a virtual env and install
>    > the current RC and see if anything important breaks, in the
>    Java/Scala
>    > you can add the staging repository to your projects resolvers and
>    test
>    > with the RC (make sure to clean up the artifact cache before/after so
>    > you don't end up building with an out of date RC going forward).
>    >
>    > ===========================================
>    > What should happen to JIRA tickets still targeting 2.4.7?
>    > ===========================================
>    >
>    > The current list of open tickets targeted at 2.4.7 can be found at:
>    > *https://issues.apache.org/jira/projects/SPARK*
>    <https://issues.apache.org/jira/projects/SPARK> and search for "Target
>    Version/s" = 2.4.7
>    >
>    > Committers should look at those and triage. Extremely important bug
>    > fixes, documentation, and API tweaks that impact compatibility should
>    > be worked on immediately. Everything else please retarget to an
>    > appropriate release.
>    >
>    > ==================
>    > But my bug isn't fixed?
>    > ==================
>    >
>    > In order to make timely releases, we will typically not hold the
>    > release unless the bug in question is a regression from the previous
>    > release. That being said, if there is something which is a regression
>    > that has not been correctly targeted please ping me or a committer to
>    > help target the issue.
>
>    ---------------------------------------------------------------------
>    To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>    <de...@spark.apache.org>
>
>
>    --
>    Twitter: *https://twitter.com/holdenkarau*
>    <https://twitter.com/holdenkarau>
>    Books (Learning Spark, High Performance Spark, etc.): *https://amzn.to/2MaRAG9
>    * <https://amzn.to/2MaRAG9>
>    YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>    <https://www.youtube.com/user/holdenkarau>
>
>    --
>    ---
>    Takeshi Yamamuro
>
>    --
> <https://databricks.com/sparkaisummit/north-america>
>
>
>    ---------------------------------------------------------------------
>    To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
>

RE: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Nicholas Marion <nm...@us.ibm.com>.
Hello,

I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there
is an active vote going through for it?


                                                                                   
 Regards,                                                                          
                                                                                   
 NICHOLAS T. MARION                                                                
 IBM Open Data Analytics for z/OS - CPO and Service Team Lead                      
                                                                                   
                                                                                     
                                                                                     
                                                                                     
 Phone: 1-845-433-5010 | Tie-Line: 293-5010                                      IBM 
 E-mail: nmarion@us.ibm.com                                                          
 Find me on: LinkedIn:                                                 2455 South Rd 
 http://www.linkedin.com/in/nicholasmarion          Poughkeepie, New York 12601-5400 
                                                                                     
                                                                       United States 
                                                                                     
                                                                                     
                                                                                     
 IBM Redbooks Silver Author                                                          
 Data Science Foundations - Level 1                                                  
                                                                                     
                                                                                     








From:	Tom Graves <tg...@yahoo.com.INVALID>
To:	Nicholas Marion <nm...@us.ibm.com>, Wenchen Fan
            <cl...@gmail.com>
Cc:	Xiao Li <li...@databricks.com>, dev <de...@spark.apache.org>,
            Takeshi Yamamuro <li...@gmail.com>, Prashant Sharma
            <sc...@gmail.com>
Date:	08/21/2020 03:09 PM
Subject:	[EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)



There is a correctness issue with caching that should go into this if
possible: https://github.com/apache/spark/pull/29506

Tom

On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan
<cl...@gmail.com> wrote:


I think so. I don't see other bug reports for 2.4.

On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <nm...@us.ibm.com>
wrote:
  It appears all 3 issues slated for Spark 2.4.7 have been merged. Should
  we be looking at getting RC2 ready?




                                                                                   
 Regards,                                                                          
                                                                                   
 NICHOLAS T. MARION                                                                
 IBM Open Data Analytics for z/OS - CPO and Service Team Lead                      
                                                                                   


                                                                                    
                                                                                    
                                                                                    
 Phone: 1-845-433-5010 | Tie-Line: 293-5010                                         
 E-mail: nmarion@us.ibm.com                                                         
 Find me on:                                                          2455 South Rd 
                                                              Poughkeepie, New York 
                                                                         12601-5400 
                                                                      United States 
                                                                                    
                                                                                    
                                                                                    
                                                                                    
                                                                                    





  Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI
  REMOVEDXiao Li ---08/17/2020 11:33:30 AM---
  https://issues.apache.org/jira/browse/SPARK-32609

  From: Xiao Li <li...@databricks.com>
  To: Prashant Sharma <sc...@gmail.com>
  Cc: Takeshi Yamamuro <li...@gmail.com>, dev <de...@spark.apache.org>
  Date: 08/17/2020 11:33 AM
  Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)



  https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to
  fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
  upcoming Spark 2.4.7 release.

  Thanks,

  Xiao

  On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <sc...@gmail.com>
  wrote:
  Thanks for letting us know. So this vote is cancelled in favor of RC2.



  On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <li...@gmail.com>
  wrote:
  Thanks for letting us know about the two issues above, Dongjoon.

  ----
  I've checked the release materials (signatures, tag, ...) and it looks
  fine, too.
  Also, I run the tests on my local Mac (java 1.8.0) with the options
  `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
  -Psparkr`
  and they passed.

  Bests,
  Takeshi



  On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <do...@gmail.com>
  wrote:
  Another instance is SPARK-31703 which filed on May 13th and the PR
  arrived two days ago.

      [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
  big endian platforms
      https://github.com/apache/spark/pull/29383

  It seems that the patch is already ready in this case.
  I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark
  2.4.7 and 3.0.1.

  Bests,
  Dongjoon.


  On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca> wrote:
  I'm going to go ahead and vote -0 then based on that then.

  On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com>
  wrote:
  Hi, All.

  Unfortunately, there is an on-going discussion about the new decimal
  correctness.

  Although we fixed one correctness issue at master and backported it
  partially to 3.0/2.4, it turns out that it needs more patched to be
  complete.

  Please see https://github.com/apache/spark/pull/29125 for on-going
  discussion for both 3.0/2.4.

      [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
  overflowed value

  I also confirmed that 2.4.7 RC1 is affected.

  Bests,
  Dongjoon.


  On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
  +1 from me. The same as usual. Licenses and sigs look OK, builds and
  passes tests on a standard selection of profiles.

  On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
  wrote:
  >
  > Please vote on releasing the following candidate as Apache Spark
  version 2.4.7.
  >
  > The vote is open until Aug 9th at 9AM PST and passes if a majority +1
  PMC votes are cast, with a minimum of 3 +1 votes.
  >
  > [ ] +1 Release this package as Apache Spark 2.4.7
  > [ ] -1 Do not release this package because ...
  >
  > To learn more about Apache Spark, please see http://spark.apache.org/
  >
  > There are currently no issues targeting 2.4.7 (try project = SPARK AND
  "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
  Progress"))
  >
  > The tag to be voted on is v2.4.7-rc1 (commit
  dc04bf53fe821b7a07f817966c6c173f3b3788c6):
  > https://github.com/apache/spark/tree/v2.4.7-rc1
  >
  > The release files, including signatures, digests, etc. can be found at:
  > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
  >
  > Signatures used for Spark RCs can be found in this file:
  > https://dist.apache.org/repos/dist/dev/spark/KEYS
  >
  > The staging repository for this release can be found at:
  > https://repository.apache.org/content/repositories/orgapachespark-1352/
  >
  > The documentation corresponding to this release can be found at:
  > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
  >
  > The list of bug fixes going into 2.4.7 can be found at the following
  URL:
  > https://s.apache.org/spark-v2.4.7-rc1
  >
  > This release is using the release script of the tag v2.4.7-rc1.
  >
  > FAQ
  >
  >
  > =========================
  > How can I help test this release?
  > =========================
  >
  > If you are a Spark user, you can help us test this release by taking
  > an existing Spark workload and running on this release candidate, then
  > reporting any regressions.
  >
  > If you're working in PySpark you can set up a virtual env and install
  > the current RC and see if anything important breaks, in the Java/Scala
  > you can add the staging repository to your projects resolvers and test
  > with the RC (make sure to clean up the artifact cache before/after so
  > you don't end up building with an out of date RC going forward).
  >
  > ===========================================
  > What should happen to JIRA tickets still targeting 2.4.7?
  > ===========================================
  >
  > The current list of open tickets targeted at 2.4.7 can be found at:
  > https://issues.apache.org/jira/projects/SPARK and search for "Target
  Version/s" = 2.4.7
  >
  > Committers should look at those and triage. Extremely important bug
  > fixes, documentation, and API tweaks that impact compatibility should
  > be worked on immediately. Everything else please retarget to an
  > appropriate release.
  >
  > ==================
  > But my bug isn't fixed?
  > ==================
  >
  > In order to make timely releases, we will typically not hold the
  > release unless the bug in question is a regression from the previous
  > release. That being said, if there is something which is a regression
  > that has not been correctly targeted please ping me or a committer to
  > help target the issue.

  ---------------------------------------------------------------------
  To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


  --
  Twitter: https://twitter.com/holdenkarau
  Books (Learning Spark, High Performance Spark, etc.):
  https://amzn.to/2MaRAG9
  YouTube Live Streams: https://www.youtube.com/user/holdenkarau

  --
  ---
  Takeshi Yamamuro

  --



  ---------------------------------------------------------------------
  To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Tom Graves <tg...@yahoo.com.INVALID>.
 There is a correctness issue with caching that should go into this if possible: https://github.com/apache/spark/pull/29506
Tom
    On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <cl...@gmail.com> wrote:  
 
 I think so. I don't see other bug reports for 2.4.
On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <nm...@us.ibm.com> wrote:


It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?




| Regards, 

NICHOLAS T. MARION 
IBM Open Data Analytics for z/OS - CPO and Service Team Lead  |


|  |
| Phone: 1-845-433-5010 | Tie-Line: 293-5010 
E-mail: nmarion@us.ibm.com 
Find me on:   | 

2455 South Rd 
Poughkeepie, New York 12601-5400 
United States  |
|  |
|  |




Xiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

From: Xiao Li <li...@databricks.com>
To: Prashant Sharma <sc...@gmail.com>
Cc: Takeshi Yamamuro <li...@gmail.com>, dev <de...@spark.apache.org>
Date: 08/17/2020 11:33 AM
Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)





https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release. 

Thanks,

Xiao

On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <sc...@gmail.com> wrote:   
Thanks for letting us know. So this vote is cancelled in favor of RC2.



On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <li...@gmail.com> wrote:
Thanks for letting us know about the two issues above, Dongjoon.

----
I've checked the release materials (signatures, tag, ...) and it looks fine, too.
Also, I run the tests on my local Mac (java 1.8.0) with the options
`-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
and they passed.

Bests,
Takeshi



On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <do...@gmail.com> wrote:   
Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
    https://github.com/apache/spark/pull/29383

It seems that the patch is already ready in this case.
I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

Bests,
Dongjoon.


On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca> wrote:
I'm going to go ahead and vote -0 then based on that then.

On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com> wrote:   
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal correctness.

Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

-- 
---
Takeshi Yamamuro

-- 






  

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Wenchen Fan <cl...@gmail.com>.
I think so. I don't see other bug reports for 2.4.

On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <nm...@us.ibm.com> wrote:

> It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we
> be looking at getting RC2 ready?
>
>
> Regards,
>
> *NICHOLAS T. MARION *
> IBM Open Data Analytics for z/OS - *CPO* and *Service Team Lead*
> ------------------------------
> *Phone: *1-845-433-5010 | *Tie-Line: *293-5010
> *E-mail:* *nmarion@us.ibm.com* <nm...@us.ibm.com>
> *Find me on:* <http://www.linkedin.com/in/nicholasmarion>
>
>
> 2455 South Rd
> Poughkeepie, New York 12601-5400
> United States
>
>
>
> [image: Inactive hide details for Xiao Li ---08/17/2020 11:33:30
> AM---INVALID URI REMOVED]Xiao Li ---08/17/2020 11:33:30 AM---
> https://issues.apache.org/jira/browse/SPARK-32609
>
> From: Xiao Li <li...@databricks.com>
> To: Prashant Sharma <sc...@gmail.com>
> Cc: Takeshi Yamamuro <li...@gmail.com>, dev <de...@spark.apache.org>
> Date: 08/17/2020 11:33 AM
> Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)
>
> ------------------------------
>
>
>
> *https://issues.apache.org/jira/browse/SPARK-32609*
> <https://issues.apache.org/jira/browse/SPARK-32609> got merged. This is
> to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
> upcoming Spark 2.4.7 release.
>
> Thanks,
>
> Xiao
>
> On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <*scrapcodes@gmail.com*
> <sc...@gmail.com>> wrote:
>
>    Thanks for letting us know. So this vote is cancelled in favor of RC2.
>
>
>
>    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
>    *linguin.m.s@gmail.com* <li...@gmail.com>> wrote:
>    Thanks for letting us know about the two issues above, Dongjoon.
>
>    ----
>    I've checked the release materials (signatures, tag, ...) and it looks
>    fine, too.
>    Also, I run the tests on my local Mac (java 1.8.0) with the options
>    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>    -Psparkr`
>    and they passed.
>
>    Bests,
>    Takeshi
>
>
>
>    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
>    *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>       Another instance is SPARK-31703 which filed on May 13th and the PR
>       arrived two days ago.
>
>           [SPARK-31703][SQL] Parquet RLE float/double are read
>       incorrectly on big endian platforms
>           *https://github.com/apache/spark/pull/29383*
>       <https://github.com/apache/spark/pull/29383>
>
>       It seems that the patch is already ready in this case.
>       I raised the priority of SPARK-31703 to `Blocker` for both Apache
>       Spark 2.4.7 and 3.0.1.
>
>       Bests,
>       Dongjoon.
>
>
>       On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <*holden@pigscanfly.ca*
>       <ho...@pigscanfly.ca>> wrote:
>       I'm going to go ahead and vote -0 then based on that then.
>
>       On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
>       *dongjoon.hyun@gmail.com* <do...@gmail.com>> wrote:
>          Hi, All.
>
>          Unfortunately, there is an on-going discussion about the new
>          decimal correctness.
>
>          Although we fixed one correctness issue at master and backported
>          it partially to 3.0/2.4, it turns out that it needs more patched to be
>          complete.
>
>          Please see *https://github.com/apache/spark/pull/29125*
>          <https://github.com/apache/spark/pull/29125> for on-going
>          discussion for both 3.0/2.4.
>
>              [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null
>          with overflowed value
>
>          I also confirmed that 2.4.7 RC1 is affected.
>
>          Bests,
>          Dongjoon.
>
>
>          On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <*srowen@apache.org*
>          <sr...@apache.org>> wrote:
>          +1 from me. The same as usual. Licenses and sigs look OK, builds
>          and
>          passes tests on a standard selection of profiles.
>
>          On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
>          *scrapcodes@gmail.com* <sc...@gmail.com>> wrote:
>          >
>          > Please vote on releasing the following candidate as Apache
>          Spark version 2.4.7.
>          >
>          > The vote is open until Aug 9th at 9AM PST and passes if a
>          majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>          >
>          > [ ] +1 Release this package as Apache Spark 2.4.7
>          > [ ] -1 Do not release this package because ...
>          >
>          > To learn more about Apache Spark, please see
>          *http://spark.apache.org/* <http://spark.apache.org/>
>          >
>          > There are currently no issues targeting 2.4.7 (try project =
>          SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>          Progress"))
>          >
>          > The tag to be voted on is v2.4.7-rc1 (commit
>          dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>          > *https://github.com/apache/spark/tree/v2.4.7-rc1*
>          <https://github.com/apache/spark/tree/v2.4.7-rc1>
>          >
>          > The release files, including signatures, digests, etc. can be
>          found at:
>          > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/*
>          <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/>
>          >
>          > Signatures used for Spark RCs can be found in this file:
>          > *https://dist.apache.org/repos/dist/dev/spark/KEYS*
>          <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>          >
>          > The staging repository for this release can be found at:
>          >
>          *https://repository.apache.org/content/repositories/orgapachespark-1352/*
>          <https://repository.apache.org/content/repositories/orgapachespark-1352/>
>          >
>          > The documentation corresponding to this release can be found
>          at:
>          > *https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/*
>          <https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/>
>          >
>          > The list of bug fixes going into 2.4.7 can be found at the
>          following URL:
>          > *https://s.apache.org/spark-v2.4.7-rc1*
>          <https://s.apache.org/spark-v2.4.7-rc1>
>          >
>          > This release is using the release script of the tag v2.4.7-rc1.
>          >
>          > FAQ
>          >
>          >
>          > =========================
>          > How can I help test this release?
>          > =========================
>          >
>          > If you are a Spark user, you can help us test this release by
>          taking
>          > an existing Spark workload and running on this release
>          candidate, then
>          > reporting any regressions.
>          >
>          > If you're working in PySpark you can set up a virtual env and
>          install
>          > the current RC and see if anything important breaks, in the
>          Java/Scala
>          > you can add the staging repository to your projects resolvers
>          and test
>          > with the RC (make sure to clean up the artifact cache
>          before/after so
>          > you don't end up building with an out of date RC going
>          forward).
>          >
>          > ===========================================
>          > What should happen to JIRA tickets still targeting 2.4.7?
>          > ===========================================
>          >
>          > The current list of open tickets targeted at 2.4.7 can be
>          found at:
>          > *https://issues.apache.org/jira/projects/SPARK*
>          <https://issues.apache.org/jira/projects/SPARK> and search for
>          "Target Version/s" = 2.4.7
>          >
>          > Committers should look at those and triage. Extremely
>          important bug
>          > fixes, documentation, and API tweaks that impact compatibility
>          should
>          > be worked on immediately. Everything else please retarget to an
>          > appropriate release.
>          >
>          > ==================
>          > But my bug isn't fixed?
>          > ==================
>          >
>          > In order to make timely releases, we will typically not hold
>          the
>          > release unless the bug in question is a regression from the
>          previous
>          > release. That being said, if there is something which is a
>          regression
>          > that has not been correctly targeted please ping me or a
>          committer to
>          > help target the issue.
>
>
>          ---------------------------------------------------------------------
>          To unsubscribe e-mail: *dev-unsubscribe@spark.apache.org*
>          <de...@spark.apache.org>
>
>
>       --
>       Twitter: *https://twitter.com/holdenkarau*
>       <https://twitter.com/holdenkarau>
>       Books (Learning Spark, High Performance Spark, etc.):
>       *https://amzn.to/2MaRAG9 * <https://amzn.to/2MaRAG9>
>       YouTube Live Streams: *https://www.youtube.com/user/holdenkarau*
>       <https://www.youtube.com/user/holdenkarau>
>
>
>    --
>    ---
>    Takeshi Yamamuro
>
>
>
> --
> <https://databricks.com/sparkaisummit/north-america>
>
>
>

RE: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Nicholas Marion <nm...@us.ibm.com>.
It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we
be looking at getting RC2 ready?


                                                                                   
 Regards,                                                                          
                                                                                   
 NICHOLAS T. MARION                                                                
 IBM Open Data Analytics for z/OS - CPO and Service Team Lead                      
                                                                                   
                                                                                     
                                                                                     
                                                                                     
 Phone: 1-845-433-5010 | Tie-Line: 293-5010                                          
 E-mail: nmarion@us.ibm.com                                                          
 Find me on:                                                           2455 South Rd 
                                                    Poughkeepie, New York 12601-5400 
                                                                                     
                                                                       United States 
                                                                                     
                                                                                     
                                                                                     
                                                                                     
                                                                                     








From:	Xiao Li <li...@databricks.com>
To:	Prashant Sharma <sc...@gmail.com>
Cc:	Takeshi Yamamuro <li...@gmail.com>, dev
            <de...@spark.apache.org>
Date:	08/17/2020 11:33 AM
Subject:	[EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)



https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to
fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
upcoming Spark 2.4.7 release.

Thanks,

Xiao

On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <sc...@gmail.com>
wrote:
  Thanks for letting us know. So this vote is cancelled in favor of RC2.



  On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <li...@gmail.com>
  wrote:
   Thanks for letting us know about the two issues above, Dongjoon.

   ----
   I've checked the release materials (signatures, tag, ...) and it looks
   fine, too.
   Also, I run the tests on my local Mac (java 1.8.0) with the options
   `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
   -Psparkr`
   and they passed.

   Bests,
   Takeshi



   On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <do...@gmail.com>
   wrote:
     Another instance is SPARK-31703 which filed on May 13th and the PR
     arrived two days ago.

         [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly
     on big endian platforms
         https://github.com/apache/spark/pull/29383

     It seems that the patch is already ready in this case.
     I raised the priority of SPARK-31703 to `Blocker` for both Apache
     Spark 2.4.7 and 3.0.1.

     Bests,
     Dongjoon.


     On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca>
     wrote:
      I'm going to go ahead and vote -0 then based on that then.

      On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
      dongjoon.hyun@gmail.com> wrote:
        Hi, All.

        Unfortunately, there is an on-going discussion about the new
        decimal correctness.

        Although we fixed one correctness issue at master and backported it
        partially to 3.0/2.4, it turns out that it needs more patched to be
        complete.

        Please see https://github.com/apache/spark/pull/29125 for on-going
        discussion for both 3.0/2.4.

            [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null
        with overflowed value

        I also confirmed that 2.4.7 RC1 is affected.

        Bests,
        Dongjoon.


        On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
          +1 from me. The same as usual. Licenses and sigs look OK, builds
          and
          passes tests on a standard selection of profiles.

          On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
          scrapcodes@gmail.com> wrote:
          >
          > Please vote on releasing the following candidate as Apache
          Spark version 2.4.7.
          >
          > The vote is open until Aug 9th at 9AM PST and passes if a
          majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
          >
          > [ ] +1 Release this package as Apache Spark 2.4.7
          > [ ] -1 Do not release this package because ...
          >
          > To learn more about Apache Spark, please see
          http://spark.apache.org/
          >
          > There are currently no issues targeting 2.4.7 (try project =
          SPARK AND "Target Version/s" = "2.4.7" AND status in (Open,
          Reopened, "In Progress"))
          >
          > The tag to be voted on is v2.4.7-rc1 (commit
          dc04bf53fe821b7a07f817966c6c173f3b3788c6):
          > https://github.com/apache/spark/tree/v2.4.7-rc1
          >
          > The release files, including signatures, digests, etc. can be
          found at:
          > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
          >
          > Signatures used for Spark RCs can be found in this file:
          > https://dist.apache.org/repos/dist/dev/spark/KEYS
          >
          > The staging repository for this release can be found at:
          >
          https://repository.apache.org/content/repositories/orgapachespark-1352/

          >
          > The documentation corresponding to this release can be found
          at:
          > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
          >
          > The list of bug fixes going into 2.4.7 can be found at the
          following URL:
          > https://s.apache.org/spark-v2.4.7-rc1
          >
          > This release is using the release script of the tag v2.4.7-rc1.
          >
          > FAQ
          >
          >
          > =========================
          > How can I help test this release?
          > =========================
          >
          > If you are a Spark user, you can help us test this release by
          taking
          > an existing Spark workload and running on this release
          candidate, then
          > reporting any regressions.
          >
          > If you're working in PySpark you can set up a virtual env and
          install
          > the current RC and see if anything important breaks, in the
          Java/Scala
          > you can add the staging repository to your projects resolvers
          and test
          > with the RC (make sure to clean up the artifact cache
          before/after so
          > you don't end up building with an out of date RC going
          forward).
          >
          > ===========================================
          > What should happen to JIRA tickets still targeting 2.4.7?
          > ===========================================
          >
          > The current list of open tickets targeted at 2.4.7 can be found
          at:
          > https://issues.apache.org/jira/projects/SPARK and search for
          "Target Version/s" = 2.4.7
          >
          > Committers should look at those and triage. Extremely important
          bug
          > fixes, documentation, and API tweaks that impact compatibility
          should
          > be worked on immediately. Everything else please retarget to an
          > appropriate release.
          >
          > ==================
          > But my bug isn't fixed?
          > ==================
          >
          > In order to make timely releases, we will typically not hold
          the
          > release unless the bug in question is a regression from the
          previous
          > release. That being said, if there is something which is a
          regression
          > that has not been correctly targeted please ping me or a
          committer to
          > help target the issue.

          ---------------------------------------------------------------------

          To unsubscribe e-mail: dev-unsubscribe@spark.apache.org



      --
      Twitter: https://twitter.com/holdenkarau
      Books (Learning Spark, High Performance Spark, etc.):
      https://amzn.to/2MaRAG9
      YouTube Live Streams: https://www.youtube.com/user/holdenkarau


   --
   ---
   Takeshi Yamamuro


--



Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Xiao Li <li...@databricks.com>.
https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to
fix a correctness bug in DSV2 of Spark 2.4. Please include it in the
upcoming Spark 2.4.7 release.

Thanks,

Xiao

On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <sc...@gmail.com>
wrote:

> Thanks for letting us know. So this vote is cancelled in favor of RC2.
>
>
>
> On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <li...@gmail.com>
> wrote:
>
>> Thanks for letting us know about the two issues above, Dongjoon.
>>
>> ----
>> I've checked the release materials (signatures, tag, ...) and it looks
>> fine, too.
>> Also, I run the tests on my local Mac (java 1.8.0) with the options
>> `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
>> -Psparkr`
>> and they passed.
>>
>> Bests,
>> Takeshi
>>
>>
>>
>> On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Another instance is SPARK-31703 which filed on May 13th and the PR
>>> arrived two days ago.
>>>
>>>     [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>>> big endian platforms
>>>     https://github.com/apache/spark/pull/29383
>>>
>>> It seems that the patch is already ready in this case.
>>> I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark
>>> 2.4.7 and 3.0.1.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca>
>>> wrote:
>>>
>>>> I'm going to go ahead and vote -0 then based on that then.
>>>>
>>>> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, All.
>>>>>
>>>>> Unfortunately, there is an on-going discussion about the new decimal
>>>>> correctness.
>>>>>
>>>>> Although we fixed one correctness issue at master and backported it
>>>>> partially to 3.0/2.4, it turns out that it needs more patched to be
>>>>> complete.
>>>>>
>>>>> Please see https://github.com/apache/spark/pull/29125 for on-going
>>>>> discussion for both 3.0/2.4.
>>>>>
>>>>>     [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>>>> overflowed value
>>>>>
>>>>> I also confirmed that 2.4.7 RC1 is affected.
>>>>>
>>>>> Bests,
>>>>> Dongjoon.
>>>>>
>>>>>
>>>>> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
>>>>>
>>>>>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>>>>> passes tests on a standard selection of profiles.
>>>>>>
>>>>>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Please vote on releasing the following candidate as Apache Spark
>>>>>> version 2.4.7.
>>>>>> >
>>>>>> > The vote is open until Aug 9th at 9AM PST and passes if a majority
>>>>>> +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>>>> >
>>>>>> > [ ] +1 Release this package as Apache Spark 2.4.7
>>>>>> > [ ] -1 Do not release this package because ...
>>>>>> >
>>>>>> > To learn more about Apache Spark, please see
>>>>>> http://spark.apache.org/
>>>>>> >
>>>>>> > There are currently no issues targeting 2.4.7 (try project = SPARK
>>>>>> AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>>>>> Progress"))
>>>>>> >
>>>>>> > The tag to be voted on is v2.4.7-rc1 (commit
>>>>>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>>>>> > https://github.com/apache/spark/tree/v2.4.7-rc1
>>>>>> >
>>>>>> > The release files, including signatures, digests, etc. can be found
>>>>>> at:
>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>>>>>> >
>>>>>> > Signatures used for Spark RCs can be found in this file:
>>>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>> >
>>>>>> > The staging repository for this release can be found at:
>>>>>> >
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1352/
>>>>>> >
>>>>>> > The documentation corresponding to this release can be found at:
>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>>>>>> >
>>>>>> > The list of bug fixes going into 2.4.7 can be found at the
>>>>>> following URL:
>>>>>> > https://s.apache.org/spark-v2.4.7-rc1
>>>>>> >
>>>>>> > This release is using the release script of the tag v2.4.7-rc1.
>>>>>> >
>>>>>> > FAQ
>>>>>> >
>>>>>> >
>>>>>> > =========================
>>>>>> > How can I help test this release?
>>>>>> > =========================
>>>>>> >
>>>>>> > If you are a Spark user, you can help us test this release by taking
>>>>>> > an existing Spark workload and running on this release candidate,
>>>>>> then
>>>>>> > reporting any regressions.
>>>>>> >
>>>>>> > If you're working in PySpark you can set up a virtual env and
>>>>>> install
>>>>>> > the current RC and see if anything important breaks, in the
>>>>>> Java/Scala
>>>>>> > you can add the staging repository to your projects resolvers and
>>>>>> test
>>>>>> > with the RC (make sure to clean up the artifact cache before/after
>>>>>> so
>>>>>> > you don't end up building with an out of date RC going forward).
>>>>>> >
>>>>>> > ===========================================
>>>>>> > What should happen to JIRA tickets still targeting 2.4.7?
>>>>>> > ===========================================
>>>>>> >
>>>>>> > The current list of open tickets targeted at 2.4.7 can be found at:
>>>>>> > https://issues.apache.org/jira/projects/SPARK and search for
>>>>>> "Target Version/s" = 2.4.7
>>>>>> >
>>>>>> > Committers should look at those and triage. Extremely important bug
>>>>>> > fixes, documentation, and API tweaks that impact compatibility
>>>>>> should
>>>>>> > be worked on immediately. Everything else please retarget to an
>>>>>> > appropriate release.
>>>>>> >
>>>>>> > ==================
>>>>>> > But my bug isn't fixed?
>>>>>> > ==================
>>>>>> >
>>>>>> > In order to make timely releases, we will typically not hold the
>>>>>> > release unless the bug in question is a regression from the previous
>>>>>> > release. That being said, if there is something which is a
>>>>>> regression
>>>>>> > that has not been correctly targeted please ping me or a committer
>>>>>> to
>>>>>> > help target the issue.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>

-- 
<https://databricks.com/sparkaisummit/north-america>

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Prashant Sharma <sc...@gmail.com>.
Thanks for letting us know. So this vote is cancelled in favor of RC2.



On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <li...@gmail.com>
wrote:

> Thanks for letting us know about the two issues above, Dongjoon.
>
> ----
> I've checked the release materials (signatures, tag, ...) and it looks
> fine, too.
> Also, I run the tests on my local Mac (java 1.8.0) with the options
> `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
> -Psparkr`
> and they passed.
>
> Bests,
> Takeshi
>
>
>
> On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Another instance is SPARK-31703 which filed on May 13th and the PR
>> arrived two days ago.
>>
>>     [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
>> big endian platforms
>>     https://github.com/apache/spark/pull/29383
>>
>> It seems that the patch is already ready in this case.
>> I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark
>> 2.4.7 and 3.0.1.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>>> I'm going to go ahead and vote -0 then based on that then.
>>>
>>> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> Hi, All.
>>>>
>>>> Unfortunately, there is an on-going discussion about the new decimal
>>>> correctness.
>>>>
>>>> Although we fixed one correctness issue at master and backported it
>>>> partially to 3.0/2.4, it turns out that it needs more patched to be
>>>> complete.
>>>>
>>>> Please see https://github.com/apache/spark/pull/29125 for on-going
>>>> discussion for both 3.0/2.4.
>>>>
>>>>     [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>>> overflowed value
>>>>
>>>> I also confirmed that 2.4.7 RC1 is affected.
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>>
>>>> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
>>>>
>>>>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>>>> passes tests on a standard selection of profiles.
>>>>>
>>>>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Please vote on releasing the following candidate as Apache Spark
>>>>> version 2.4.7.
>>>>> >
>>>>> > The vote is open until Aug 9th at 9AM PST and passes if a majority
>>>>> +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>>> >
>>>>> > [ ] +1 Release this package as Apache Spark 2.4.7
>>>>> > [ ] -1 Do not release this package because ...
>>>>> >
>>>>> > To learn more about Apache Spark, please see
>>>>> http://spark.apache.org/
>>>>> >
>>>>> > There are currently no issues targeting 2.4.7 (try project = SPARK
>>>>> AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>>>> Progress"))
>>>>> >
>>>>> > The tag to be voted on is v2.4.7-rc1 (commit
>>>>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>>>> > https://github.com/apache/spark/tree/v2.4.7-rc1
>>>>> >
>>>>> > The release files, including signatures, digests, etc. can be found
>>>>> at:
>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>>>>> >
>>>>> > Signatures used for Spark RCs can be found in this file:
>>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>> >
>>>>> > The staging repository for this release can be found at:
>>>>> >
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1352/
>>>>> >
>>>>> > The documentation corresponding to this release can be found at:
>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>>>>> >
>>>>> > The list of bug fixes going into 2.4.7 can be found at the following
>>>>> URL:
>>>>> > https://s.apache.org/spark-v2.4.7-rc1
>>>>> >
>>>>> > This release is using the release script of the tag v2.4.7-rc1.
>>>>> >
>>>>> > FAQ
>>>>> >
>>>>> >
>>>>> > =========================
>>>>> > How can I help test this release?
>>>>> > =========================
>>>>> >
>>>>> > If you are a Spark user, you can help us test this release by taking
>>>>> > an existing Spark workload and running on this release candidate,
>>>>> then
>>>>> > reporting any regressions.
>>>>> >
>>>>> > If you're working in PySpark you can set up a virtual env and install
>>>>> > the current RC and see if anything important breaks, in the
>>>>> Java/Scala
>>>>> > you can add the staging repository to your projects resolvers and
>>>>> test
>>>>> > with the RC (make sure to clean up the artifact cache before/after so
>>>>> > you don't end up building with an out of date RC going forward).
>>>>> >
>>>>> > ===========================================
>>>>> > What should happen to JIRA tickets still targeting 2.4.7?
>>>>> > ===========================================
>>>>> >
>>>>> > The current list of open tickets targeted at 2.4.7 can be found at:
>>>>> > https://issues.apache.org/jira/projects/SPARK and search for
>>>>> "Target Version/s" = 2.4.7
>>>>> >
>>>>> > Committers should look at those and triage. Extremely important bug
>>>>> > fixes, documentation, and API tweaks that impact compatibility should
>>>>> > be worked on immediately. Everything else please retarget to an
>>>>> > appropriate release.
>>>>> >
>>>>> > ==================
>>>>> > But my bug isn't fixed?
>>>>> > ==================
>>>>> >
>>>>> > In order to make timely releases, we will typically not hold the
>>>>> > release unless the bug in question is a regression from the previous
>>>>> > release. That being said, if there is something which is a regression
>>>>> > that has not been correctly targeted please ping me or a committer to
>>>>> > help target the issue.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>
> --
> ---
> Takeshi Yamamuro
>

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Takeshi Yamamuro <li...@gmail.com>.
Thanks for letting us know about the two issues above, Dongjoon.

----
I've checked the release materials (signatures, tag, ...) and it looks
fine, too.
Also, I run the tests on my local Mac (java 1.8.0) with the options
`-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes
-Psparkr`
and they passed.

Bests,
Takeshi



On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <do...@gmail.com>
wrote:

> Another instance is SPARK-31703 which filed on May 13th and the PR arrived
> two days ago.
>
>     [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on
> big endian platforms
>     https://github.com/apache/spark/pull/29383
>
> It seems that the patch is already ready in this case.
> I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark
> 2.4.7 and 3.0.1.
>
> Bests,
> Dongjoon.
>
>
> On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> I'm going to go ahead and vote -0 then based on that then.
>>
>> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Hi, All.
>>>
>>> Unfortunately, there is an on-going discussion about the new decimal
>>> correctness.
>>>
>>> Although we fixed one correctness issue at master and backported it
>>> partially to 3.0/2.4, it turns out that it needs more patched to be
>>> complete.
>>>
>>> Please see https://github.com/apache/spark/pull/29125 for on-going
>>> discussion for both 3.0/2.4.
>>>
>>>     [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>>> overflowed value
>>>
>>> I also confirmed that 2.4.7 RC1 is affected.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
>>>
>>>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>>> passes tests on a standard selection of profiles.
>>>>
>>>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Please vote on releasing the following candidate as Apache Spark
>>>> version 2.4.7.
>>>> >
>>>> > The vote is open until Aug 9th at 9AM PST and passes if a majority +1
>>>> PMC votes are cast, with a minimum of 3 +1 votes.
>>>> >
>>>> > [ ] +1 Release this package as Apache Spark 2.4.7
>>>> > [ ] -1 Do not release this package because ...
>>>> >
>>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>>> >
>>>> > There are currently no issues targeting 2.4.7 (try project = SPARK
>>>> AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In
>>>> Progress"))
>>>> >
>>>> > The tag to be voted on is v2.4.7-rc1 (commit
>>>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>>> > https://github.com/apache/spark/tree/v2.4.7-rc1
>>>> >
>>>> > The release files, including signatures, digests, etc. can be found
>>>> at:
>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>>>> >
>>>> > Signatures used for Spark RCs can be found in this file:
>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>> >
>>>> > The staging repository for this release can be found at:
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachespark-1352/
>>>> >
>>>> > The documentation corresponding to this release can be found at:
>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>>>> >
>>>> > The list of bug fixes going into 2.4.7 can be found at the following
>>>> URL:
>>>> > https://s.apache.org/spark-v2.4.7-rc1
>>>> >
>>>> > This release is using the release script of the tag v2.4.7-rc1.
>>>> >
>>>> > FAQ
>>>> >
>>>> >
>>>> > =========================
>>>> > How can I help test this release?
>>>> > =========================
>>>> >
>>>> > If you are a Spark user, you can help us test this release by taking
>>>> > an existing Spark workload and running on this release candidate, then
>>>> > reporting any regressions.
>>>> >
>>>> > If you're working in PySpark you can set up a virtual env and install
>>>> > the current RC and see if anything important breaks, in the Java/Scala
>>>> > you can add the staging repository to your projects resolvers and test
>>>> > with the RC (make sure to clean up the artifact cache before/after so
>>>> > you don't end up building with an out of date RC going forward).
>>>> >
>>>> > ===========================================
>>>> > What should happen to JIRA tickets still targeting 2.4.7?
>>>> > ===========================================
>>>> >
>>>> > The current list of open tickets targeted at 2.4.7 can be found at:
>>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>>> Version/s" = 2.4.7
>>>> >
>>>> > Committers should look at those and triage. Extremely important bug
>>>> > fixes, documentation, and API tweaks that impact compatibility should
>>>> > be worked on immediately. Everything else please retarget to an
>>>> > appropriate release.
>>>> >
>>>> > ==================
>>>> > But my bug isn't fixed?
>>>> > ==================
>>>> >
>>>> > In order to make timely releases, we will typically not hold the
>>>> > release unless the bug in question is a regression from the previous
>>>> > release. That being said, if there is something which is a regression
>>>> > that has not been correctly targeted please ping me or a committer to
>>>> > help target the issue.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

-- 
---
Takeshi Yamamuro

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Dongjoon Hyun <do...@gmail.com>.
Another instance is SPARK-31703 which filed on May 13th and the PR arrived
two days ago.

    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big
endian platforms
    https://github.com/apache/spark/pull/29383

It seems that the patch is already ready in this case.
I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark
2.4.7 and 3.0.1.

Bests,
Dongjoon.


On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <ho...@pigscanfly.ca> wrote:

> I'm going to go ahead and vote -0 then based on that then.
>
> On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Hi, All.
>>
>> Unfortunately, there is an on-going discussion about the new decimal
>> correctness.
>>
>> Although we fixed one correctness issue at master and backported it
>> partially to 3.0/2.4, it turns out that it needs more patched to be
>> complete.
>>
>> Please see https://github.com/apache/spark/pull/29125 for on-going
>> discussion for both 3.0/2.4.
>>
>>     [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
>> overflowed value
>>
>> I also confirmed that 2.4.7 RC1 is affected.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
>>
>>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>>> passes tests on a standard selection of profiles.
>>>
>>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.7.
>>> >
>>> > The vote is open until Aug 9th at 9AM PST and passes if a majority +1
>>> PMC votes are cast, with a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 2.4.7
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>> >
>>> > There are currently no issues targeting 2.4.7 (try project = SPARK AND
>>> "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>>> >
>>> > The tag to be voted on is v2.4.7-rc1 (commit
>>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>>> > https://github.com/apache/spark/tree/v2.4.7-rc1
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1352/
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>>> >
>>> > The list of bug fixes going into 2.4.7 can be found at the following
>>> URL:
>>> > https://s.apache.org/spark-v2.4.7-rc1
>>> >
>>> > This release is using the release script of the tag v2.4.7-rc1.
>>> >
>>> > FAQ
>>> >
>>> >
>>> > =========================
>>> > How can I help test this release?
>>> > =========================
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC and see if anything important breaks, in the Java/Scala
>>> > you can add the staging repository to your projects resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with an out of date RC going forward).
>>> >
>>> > ===========================================
>>> > What should happen to JIRA tickets still targeting 2.4.7?
>>> > ===========================================
>>> >
>>> > The current list of open tickets targeted at 2.4.7 can be found at:
>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.7
>>> >
>>> > Committers should look at those and triage. Extremely important bug
>>> > fixes, documentation, and API tweaks that impact compatibility should
>>> > be worked on immediately. Everything else please retarget to an
>>> > appropriate release.
>>> >
>>> > ==================
>>> > But my bug isn't fixed?
>>> > ==================
>>> >
>>> > In order to make timely releases, we will typically not hold the
>>> > release unless the bug in question is a regression from the previous
>>> > release. That being said, if there is something which is a regression
>>> > that has not been correctly targeted please ping me or a committer to
>>> > help target the issue.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Holden Karau <ho...@pigscanfly.ca>.
I'm going to go ahead and vote -0 then based on that then.

On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <do...@gmail.com>
wrote:

> Hi, All.
>
> Unfortunately, there is an on-going discussion about the new decimal
> correctness.
>
> Although we fixed one correctness issue at master and backported it
> partially to 3.0/2.4, it turns out that it needs more patched to be
> complete.
>
> Please see https://github.com/apache/spark/pull/29125 for on-going
> discussion for both 3.0/2.4.
>
>     [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
> overflowed value
>
> I also confirmed that 2.4.7 RC1 is affected.
>
> Bests,
> Dongjoon.
>
>
> On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:
>
>> +1 from me. The same as usual. Licenses and sigs look OK, builds and
>> passes tests on a standard selection of profiles.
>>
>> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
>> wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.7.
>> >
>> > The vote is open until Aug 9th at 9AM PST and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.4.7
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > There are currently no issues targeting 2.4.7 (try project = SPARK AND
>> "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>> >
>> > The tag to be voted on is v2.4.7-rc1 (commit
>> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
>> > https://github.com/apache/spark/tree/v2.4.7-rc1
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1352/
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>> >
>> > The list of bug fixes going into 2.4.7 can be found at the following
>> URL:
>> > https://s.apache.org/spark-v2.4.7-rc1
>> >
>> > This release is using the release script of the tag v2.4.7-rc1.
>> >
>> > FAQ
>> >
>> >
>> > =========================
>> > How can I help test this release?
>> > =========================
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC and see if anything important breaks, in the Java/Scala
>> > you can add the staging repository to your projects resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with an out of date RC going forward).
>> >
>> > ===========================================
>> > What should happen to JIRA tickets still targeting 2.4.7?
>> > ===========================================
>> >
>> > The current list of open tickets targeted at 2.4.7 can be found at:
>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.7
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==================
>> > But my bug isn't fixed?
>> > ==================
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a regression
>> > that has not been correctly targeted please ping me or a committer to
>> > help target the issue.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Dongjoon Hyun <do...@gmail.com>.
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal
correctness.

Although we fixed one correctness issue at master and backported it
partially to 3.0/2.4, it turns out that it needs more patched to be
complete.

Please see https://github.com/apache/spark/pull/29125 for on-going
discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with
overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <sr...@apache.org> wrote:

> +1 from me. The same as usual. Licenses and sigs look OK, builds and
> passes tests on a standard selection of profiles.
>
> On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com>
> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version
> 2.4.7.
> >
> > The vote is open until Aug 9th at 9AM PST and passes if a majority +1
> PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.7
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > There are currently no issues targeting 2.4.7 (try project = SPARK AND
> "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
> >
> > The tag to be voted on is v2.4.7-rc1 (commit
> dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> > https://github.com/apache/spark/tree/v2.4.7-rc1
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1352/
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
> >
> > The list of bug fixes going into 2.4.7 can be found at the following URL:
> > https://s.apache.org/spark-v2.4.7-rc1
> >
> > This release is using the release script of the tag v2.4.7-rc1.
> >
> > FAQ
> >
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with an out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.7?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.7 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.7
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Release Spark 2.4.7 (RC1)

Posted by Sean Owen <sr...@apache.org>.
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <sc...@gmail.com> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org