You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Wenchen Fan <cl...@gmail.com> on 2018/10/10 16:46:52 UTC
[VOTE] SPARK 2.4.0 (RC3)
Please vote on releasing the following candidate as Apache Spark version
2.4.0.
The vote is open until October 1 PST and passes if a majority +1 PMC votes
are cast, with
a minimum of 3 +1 votes.
[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...
To learn more about Apache Spark, please see http://spark.apache.org/
The tag to be voted on is v2.4.0-rc3 (commit
8e4a99bd201b9204fec52580f19ae70a229ed94e):
https://github.com/apache/spark/tree/v2.4.0-rc3
The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS
The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1289
The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
The list of bug fixes going into 2.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12342385
FAQ
=========================
How can I help test this release?
=========================
If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.
If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).
===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================
The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target
Version/s" = 2.4.0
Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.
==================
But my bug isn't fixed?
==================
In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Koert Kuipers <ko...@tresata.com>.
i deployed 2.4.0 RC3 on our dev cluster and ran into issue with spark shell
and jline. there is probably a simple workaround so this is not a serious
issue but just wanted to let you know.
https://issues.apache.org/jira/browse/SPARK-25783
On Mon, Oct 15, 2018 at 4:59 PM Imran Rashid <ir...@cloudera.com.invalid>
wrote:
> I just discovered https://issues.apache.org/jira/browse/SPARK-25738 with
> some more testing. I only marked it as critical, but seems pretty bad --
> I'll defer to others opinion
>
> On Sat, Oct 13, 2018 at 4:15 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Yes. From my side, it's -1 for RC3.
>>
>> Bests,
>> Dongjoon.
>>
>> On Sat, Oct 13, 2018 at 1:24 PM Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>>> So if it's a blocker would you think this should be a -1?
>>>
>>> On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> Hi, Holden.
>>>>
>>>> Since that's a performance at 2.4.0, I marked as `Blocker` four days
>>>> ago.
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>>
>>>> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <ho...@pigscanfly.ca>
>>>> wrote:
>>>>
>>>>> Following up I just wanted to make sure this new blocker that Dongjoon
>>>>> designated is surfaced -
>>>>> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409&jql=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>>>>>
>>>>> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <ga...@gmail.com> wrote:
>>>>>
>>>>>> -1. We have two correctness bugs:
>>>>>> https://issues.apache.org/jira/browse/SPARK-25714 and
>>>>>> https://issues.apache.org/jira/browse/SPARK-25708.
>>>>>>
>>>>>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>>>>>
>>>>>> Xiao
>>>>>>
>>>>>>
>>>>>> Sean Owen <sr...@apache.org> 于2018年10月11日周四 上午9:04写道：
>>>>>>
>>>>>>> This is a legitimate question about the behavior of ScalaUDF after
>>>>>>> the
>>>>>>> change to support 2.12:
>>>>>>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>>>>>>> Not quite a blocker I think, but a potential gotcha we definitely
>>>>>>> need
>>>>>>> to highlight in release notes. There may be an argument for changing
>>>>>>> ScalaUDF again before the release. Have a look, anyone familiar with
>>>>>>> catalyst.
>>>>>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
>>>>>>> >
>>>>>>> > +1. I tested the source build against Scala 2.12 and common build
>>>>>>> > profiles. License and sigs look OK.
>>>>>>> >
>>>>>>> > No blockers; one critical:
>>>>>>> >
>>>>>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>>>>>> >
>>>>>>> > I think this one is "won't fix" though? not trying to restore the
>>>>>>> behavior?
>>>>>>> >
>>>>>>> > Other items open for 2.4.0:
>>>>>>> >
>>>>>>> > SPARK-25347 Document image data source in doc site
>>>>>>> > SPARK-25584 Document libsvm data source in doc site
>>>>>>> > SPARK-25179 Document the features that require Pyarrow 0.10
>>>>>>> > SPARK-25507 Update documents for the new features in 2.4 release
>>>>>>> > SPARK-25346 Document Spark builtin data sources
>>>>>>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>>>>>>> > SPARK-23197 Flaky test:
>>>>>>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>>>>>>> > SPARK-22809 pyspark is sensitive to imports with dots
>>>>>>> > SPARK-21030 extend hint syntax to support any expression for
>>>>>>> Python and R
>>>>>>> >
>>>>>>> > Anyone know enough to close or retarget them? they don't look
>>>>>>> critical
>>>>>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document
>>>>>>> the
>>>>>>> > features that require Pyarrow 0.10" however sounds like it could
>>>>>>> have
>>>>>>> > been important for 2.4? if not a blocker.
>>>>>>> >
>>>>>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>>>>>>> > there is some ongoing discussion there.
>>>>>>> >
>>>>>>> > I am evaluating
>>>>>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642
>>>>>>> right
>>>>>>> > now.
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > > Please vote on releasing the following candidate as Apache Spark
>>>>>>> version 2.4.0.
>>>>>>> > >
>>>>>>> > > The vote is open until October 1 PST and passes if a majority +1
>>>>>>> PMC votes are cast, with
>>>>>>> > > a minimum of 3 +1 votes.
>>>>>>> > >
>>>>>>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>>>>>>> > > [ ] -1 Do not release this package because ...
>>>>>>> > >
>>>>>>> > > To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>> > >
>>>>>>> > > The tag to be voted on is v2.4.0-rc3 (commit
>>>>>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>>>>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>>>>>>> > >
>>>>>>> > > The release files, including signatures, digests, etc. can be
>>>>>>> found at:
>>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>>>>> > >
>>>>>>> > > Signatures used for Spark RCs can be found in this file:
>>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>> > >
>>>>>>> > > The staging repository for this release can be found at:
>>>>>>> > >
>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>>>>> > >
>>>>>>> > > The documentation corresponding to this release can be found at:
>>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>>>>> > >
>>>>>>> > > The list of bug fixes going into 2.4.0 can be found at the
>>>>>>> following URL:
>>>>>>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>>>>> > >
>>>>>>> > > FAQ
>>>>>>> > >
>>>>>>> > > =========================
>>>>>>> > > How can I help test this release?
>>>>>>> > > =========================
>>>>>>> > >
>>>>>>> > > If you are a Spark user, you can help us test this release by
>>>>>>> taking
>>>>>>> > > an existing Spark workload and running on this release
>>>>>>> candidate, then
>>>>>>> > > reporting any regressions.
>>>>>>> > >
>>>>>>> > > If you're working in PySpark you can set up a virtual env and
>>>>>>> install
>>>>>>> > > the current RC and see if anything important breaks, in the
>>>>>>> Java/Scala
>>>>>>> > > you can add the staging repository to your projects resolvers
>>>>>>> and test
>>>>>>> > > with the RC (make sure to clean up the artifact cache
>>>>>>> before/after so
>>>>>>> > > you don't end up building with a out of date RC going forward).
>>>>>>> > >
>>>>>>> > > ===========================================
>>>>>>> > > What should happen to JIRA tickets still targeting 2.4.0?
>>>>>>> > > ===========================================
>>>>>>> > >
>>>>>>> > > The current list of open tickets targeted at 2.4.0 can be found
>>>>>>> at:
>>>>>>> > > https://issues.apache.org/jira/projects/SPARK and search for
>>>>>>> "Target Version/s" = 2.4.0
>>>>>>> > >
>>>>>>> > > Committers should look at those and triage. Extremely important
>>>>>>> bug
>>>>>>> > > fixes, documentation, and API tweaks that impact compatibility
>>>>>>> should
>>>>>>> > > be worked on immediately. Everything else please retarget to an
>>>>>>> > > appropriate release.
>>>>>>> > >
>>>>>>> > > ==================
>>>>>>> > > But my bug isn't fixed?
>>>>>>> > > ==================
>>>>>>> > >
>>>>>>> > > In order to make timely releases, we will typically not hold the
>>>>>>> > > release unless the bug in question is a regression from the
>>>>>>> previous
>>>>>>> > > release. That being said, if there is something which is a
>>>>>>> regression
>>>>>>> > > that has not been correctly targeted please ping me or a
>>>>>>> committer to
>>>>>>> > > help target the issue.
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>
>>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Imran Rashid <ir...@cloudera.com.INVALID>.
I just discovered https://issues.apache.org/jira/browse/SPARK-25738 with
some more testing. I only marked it as critical, but seems pretty bad --
I'll defer to others opinion
On Sat, Oct 13, 2018 at 4:15 PM Dongjoon Hyun <do...@gmail.com>
wrote:
> Yes. From my side, it's -1 for RC3.
>
> Bests,
> Dongjoon.
>
> On Sat, Oct 13, 2018 at 1:24 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> So if it's a blocker would you think this should be a -1?
>>
>> On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Hi, Holden.
>>>
>>> Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <ho...@pigscanfly.ca>
>>> wrote:
>>>
>>>> Following up I just wanted to make sure this new blocker that Dongjoon
>>>> designated is surfaced -
>>>> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409&jql=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>>>>
>>>> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <ga...@gmail.com> wrote:
>>>>
>>>>> -1. We have two correctness bugs:
>>>>> https://issues.apache.org/jira/browse/SPARK-25714 and
>>>>> https://issues.apache.org/jira/browse/SPARK-25708.
>>>>>
>>>>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>>>>
>>>>> Xiao
>>>>>
>>>>>
>>>>> Sean Owen <sr...@apache.org> 于2018年10月11日周四 上午9:04写道：
>>>>>
>>>>>> This is a legitimate question about the behavior of ScalaUDF after the
>>>>>> change to support 2.12:
>>>>>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>>>>>> Not quite a blocker I think, but a potential gotcha we definitely need
>>>>>> to highlight in release notes. There may be an argument for changing
>>>>>> ScalaUDF again before the release. Have a look, anyone familiar with
>>>>>> catalyst.
>>>>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
>>>>>> >
>>>>>> > +1. I tested the source build against Scala 2.12 and common build
>>>>>> > profiles. License and sigs look OK.
>>>>>> >
>>>>>> > No blockers; one critical:
>>>>>> >
>>>>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>>>>> >
>>>>>> > I think this one is "won't fix" though? not trying to restore the
>>>>>> behavior?
>>>>>> >
>>>>>> > Other items open for 2.4.0:
>>>>>> >
>>>>>> > SPARK-25347 Document image data source in doc site
>>>>>> > SPARK-25584 Document libsvm data source in doc site
>>>>>> > SPARK-25179 Document the features that require Pyarrow 0.10
>>>>>> > SPARK-25507 Update documents for the new features in 2.4 release
>>>>>> > SPARK-25346 Document Spark builtin data sources
>>>>>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>>>>>> > SPARK-23197 Flaky test:
>>>>>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>>>>>> > SPARK-22809 pyspark is sensitive to imports with dots
>>>>>> > SPARK-21030 extend hint syntax to support any expression for Python
>>>>>> and R
>>>>>> >
>>>>>> > Anyone know enough to close or retarget them? they don't look
>>>>>> critical
>>>>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document
>>>>>> the
>>>>>> > features that require Pyarrow 0.10" however sounds like it could
>>>>>> have
>>>>>> > been important for 2.4? if not a blocker.
>>>>>> >
>>>>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>>>>>> > there is some ongoing discussion there.
>>>>>> >
>>>>>> > I am evaluating
>>>>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642
>>>>>> right
>>>>>> > now.
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com>
>>>>>> wrote:
>>>>>> > >
>>>>>> > > Please vote on releasing the following candidate as Apache Spark
>>>>>> version 2.4.0.
>>>>>> > >
>>>>>> > > The vote is open until October 1 PST and passes if a majority +1
>>>>>> PMC votes are cast, with
>>>>>> > > a minimum of 3 +1 votes.
>>>>>> > >
>>>>>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>>>>>> > > [ ] -1 Do not release this package because ...
>>>>>> > >
>>>>>> > > To learn more about Apache Spark, please see
>>>>>> http://spark.apache.org/
>>>>>> > >
>>>>>> > > The tag to be voted on is v2.4.0-rc3 (commit
>>>>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>>>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>>>>>> > >
>>>>>> > > The release files, including signatures, digests, etc. can be
>>>>>> found at:
>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>>>> > >
>>>>>> > > Signatures used for Spark RCs can be found in this file:
>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>> > >
>>>>>> > > The staging repository for this release can be found at:
>>>>>> > >
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>>>> > >
>>>>>> > > The documentation corresponding to this release can be found at:
>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>>>> > >
>>>>>> > > The list of bug fixes going into 2.4.0 can be found at the
>>>>>> following URL:
>>>>>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>>>> > >
>>>>>> > > FAQ
>>>>>> > >
>>>>>> > > =========================
>>>>>> > > How can I help test this release?
>>>>>> > > =========================
>>>>>> > >
>>>>>> > > If you are a Spark user, you can help us test this release by
>>>>>> taking
>>>>>> > > an existing Spark workload and running on this release candidate,
>>>>>> then
>>>>>> > > reporting any regressions.
>>>>>> > >
>>>>>> > > If you're working in PySpark you can set up a virtual env and
>>>>>> install
>>>>>> > > the current RC and see if anything important breaks, in the
>>>>>> Java/Scala
>>>>>> > > you can add the staging repository to your projects resolvers and
>>>>>> test
>>>>>> > > with the RC (make sure to clean up the artifact cache
>>>>>> before/after so
>>>>>> > > you don't end up building with a out of date RC going forward).
>>>>>> > >
>>>>>> > > ===========================================
>>>>>> > > What should happen to JIRA tickets still targeting 2.4.0?
>>>>>> > > ===========================================
>>>>>> > >
>>>>>> > > The current list of open tickets targeted at 2.4.0 can be found
>>>>>> at:
>>>>>> > > https://issues.apache.org/jira/projects/SPARK and search for
>>>>>> "Target Version/s" = 2.4.0
>>>>>> > >
>>>>>> > > Committers should look at those and triage. Extremely important
>>>>>> bug
>>>>>> > > fixes, documentation, and API tweaks that impact compatibility
>>>>>> should
>>>>>> > > be worked on immediately. Everything else please retarget to an
>>>>>> > > appropriate release.
>>>>>> > >
>>>>>> > > ==================
>>>>>> > > But my bug isn't fixed?
>>>>>> > > ==================
>>>>>> > >
>>>>>> > > In order to make timely releases, we will typically not hold the
>>>>>> > > release unless the bug in question is a regression from the
>>>>>> previous
>>>>>> > > release. That being said, if there is something which is a
>>>>>> regression
>>>>>> > > that has not been correctly targeted please ping me or a
>>>>>> committer to
>>>>>> > > help target the issue.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Dongjoon Hyun <do...@gmail.com>.
Yes. From my side, it's -1 for RC3.
Bests,
Dongjoon.
On Sat, Oct 13, 2018 at 1:24 PM Holden Karau <ho...@pigscanfly.ca> wrote:
> So if it's a blocker would you think this should be a -1?
>
> On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Hi, Holden.
>>
>> Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>>> Following up I just wanted to make sure this new blocker that Dongjoon
>>> designated is surfaced -
>>> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409&jql=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>>>
>>> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <ga...@gmail.com> wrote:
>>>
>>>> -1. We have two correctness bugs:
>>>> https://issues.apache.org/jira/browse/SPARK-25714 and
>>>> https://issues.apache.org/jira/browse/SPARK-25708.
>>>>
>>>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>>>
>>>> Xiao
>>>>
>>>>
>>>> Sean Owen <sr...@apache.org> 于2018年10月11日周四 上午9:04写道：
>>>>
>>>>> This is a legitimate question about the behavior of ScalaUDF after the
>>>>> change to support 2.12:
>>>>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>>>>> Not quite a blocker I think, but a potential gotcha we definitely need
>>>>> to highlight in release notes. There may be an argument for changing
>>>>> ScalaUDF again before the release. Have a look, anyone familiar with
>>>>> catalyst.
>>>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
>>>>> >
>>>>> > +1. I tested the source build against Scala 2.12 and common build
>>>>> > profiles. License and sigs look OK.
>>>>> >
>>>>> > No blockers; one critical:
>>>>> >
>>>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>>>> >
>>>>> > I think this one is "won't fix" though? not trying to restore the
>>>>> behavior?
>>>>> >
>>>>> > Other items open for 2.4.0:
>>>>> >
>>>>> > SPARK-25347 Document image data source in doc site
>>>>> > SPARK-25584 Document libsvm data source in doc site
>>>>> > SPARK-25179 Document the features that require Pyarrow 0.10
>>>>> > SPARK-25507 Update documents for the new features in 2.4 release
>>>>> > SPARK-25346 Document Spark builtin data sources
>>>>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>>>>> > SPARK-23197 Flaky test:
>>>>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>>>>> > SPARK-22809 pyspark is sensitive to imports with dots
>>>>> > SPARK-21030 extend hint syntax to support any expression for Python
>>>>> and R
>>>>> >
>>>>> > Anyone know enough to close or retarget them? they don't look
>>>>> critical
>>>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document
>>>>> the
>>>>> > features that require Pyarrow 0.10" however sounds like it could have
>>>>> > been important for 2.4? if not a blocker.
>>>>> >
>>>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>>>>> > there is some ongoing discussion there.
>>>>> >
>>>>> > I am evaluating
>>>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642
>>>>> right
>>>>> > now.
>>>>> >
>>>>> >
>>>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com>
>>>>> wrote:
>>>>> > >
>>>>> > > Please vote on releasing the following candidate as Apache Spark
>>>>> version 2.4.0.
>>>>> > >
>>>>> > > The vote is open until October 1 PST and passes if a majority +1
>>>>> PMC votes are cast, with
>>>>> > > a minimum of 3 +1 votes.
>>>>> > >
>>>>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>>>>> > > [ ] -1 Do not release this package because ...
>>>>> > >
>>>>> > > To learn more about Apache Spark, please see
>>>>> http://spark.apache.org/
>>>>> > >
>>>>> > > The tag to be voted on is v2.4.0-rc3 (commit
>>>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>>>>> > >
>>>>> > > The release files, including signatures, digests, etc. can be
>>>>> found at:
>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>>> > >
>>>>> > > Signatures used for Spark RCs can be found in this file:
>>>>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>> > >
>>>>> > > The staging repository for this release can be found at:
>>>>> > >
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>>> > >
>>>>> > > The documentation corresponding to this release can be found at:
>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>>> > >
>>>>> > > The list of bug fixes going into 2.4.0 can be found at the
>>>>> following URL:
>>>>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>>> > >
>>>>> > > FAQ
>>>>> > >
>>>>> > > =========================
>>>>> > > How can I help test this release?
>>>>> > > =========================
>>>>> > >
>>>>> > > If you are a Spark user, you can help us test this release by
>>>>> taking
>>>>> > > an existing Spark workload and running on this release candidate,
>>>>> then
>>>>> > > reporting any regressions.
>>>>> > >
>>>>> > > If you're working in PySpark you can set up a virtual env and
>>>>> install
>>>>> > > the current RC and see if anything important breaks, in the
>>>>> Java/Scala
>>>>> > > you can add the staging repository to your projects resolvers and
>>>>> test
>>>>> > > with the RC (make sure to clean up the artifact cache before/after
>>>>> so
>>>>> > > you don't end up building with a out of date RC going forward).
>>>>> > >
>>>>> > > ===========================================
>>>>> > > What should happen to JIRA tickets still targeting 2.4.0?
>>>>> > > ===========================================
>>>>> > >
>>>>> > > The current list of open tickets targeted at 2.4.0 can be found at:
>>>>> > > https://issues.apache.org/jira/projects/SPARK and search for
>>>>> "Target Version/s" = 2.4.0
>>>>> > >
>>>>> > > Committers should look at those and triage. Extremely important bug
>>>>> > > fixes, documentation, and API tweaks that impact compatibility
>>>>> should
>>>>> > > be worked on immediately. Everything else please retarget to an
>>>>> > > appropriate release.
>>>>> > >
>>>>> > > ==================
>>>>> > > But my bug isn't fixed?
>>>>> > > ==================
>>>>> > >
>>>>> > > In order to make timely releases, we will typically not hold the
>>>>> > > release unless the bug in question is a regression from the
>>>>> previous
>>>>> > > release. That being said, if there is something which is a
>>>>> regression
>>>>> > > that has not been correctly targeted please ping me or a committer
>>>>> to
>>>>> > > help target the issue.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Holden Karau <ho...@pigscanfly.ca>.
So if it's a blocker would you think this should be a -1?
On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <do...@gmail.com>
wrote:
> Hi, Holden.
>
> Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.
>
> Bests,
> Dongjoon.
>
>
> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <ho...@pigscanfly.ca>
> wrote:
>
>> Following up I just wanted to make sure this new blocker that Dongjoon
>> designated is surfaced -
>> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409&jql=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>>
>> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <ga...@gmail.com> wrote:
>>
>>> -1. We have two correctness bugs:
>>> https://issues.apache.org/jira/browse/SPARK-25714 and
>>> https://issues.apache.org/jira/browse/SPARK-25708.
>>>
>>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>>
>>> Xiao
>>>
>>>
>>> Sean Owen <sr...@apache.org> 于2018年10月11日周四 上午9:04写道：
>>>
>>>> This is a legitimate question about the behavior of ScalaUDF after the
>>>> change to support 2.12:
>>>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>>>> Not quite a blocker I think, but a potential gotcha we definitely need
>>>> to highlight in release notes. There may be an argument for changing
>>>> ScalaUDF again before the release. Have a look, anyone familiar with
>>>> catalyst.
>>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
>>>> >
>>>> > +1. I tested the source build against Scala 2.12 and common build
>>>> > profiles. License and sigs look OK.
>>>> >
>>>> > No blockers; one critical:
>>>> >
>>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>>> >
>>>> > I think this one is "won't fix" though? not trying to restore the
>>>> behavior?
>>>> >
>>>> > Other items open for 2.4.0:
>>>> >
>>>> > SPARK-25347 Document image data source in doc site
>>>> > SPARK-25584 Document libsvm data source in doc site
>>>> > SPARK-25179 Document the features that require Pyarrow 0.10
>>>> > SPARK-25507 Update documents for the new features in 2.4 release
>>>> > SPARK-25346 Document Spark builtin data sources
>>>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>>>> > SPARK-23197 Flaky test:
>>>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>>>> > SPARK-22809 pyspark is sensitive to imports with dots
>>>> > SPARK-21030 extend hint syntax to support any expression for Python
>>>> and R
>>>> >
>>>> > Anyone know enough to close or retarget them? they don't look critical
>>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
>>>> > features that require Pyarrow 0.10" however sounds like it could have
>>>> > been important for 2.4? if not a blocker.
>>>> >
>>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>>>> > there is some ongoing discussion there.
>>>> >
>>>> > I am evaluating
>>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642
>>>> right
>>>> > now.
>>>> >
>>>> >
>>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com>
>>>> wrote:
>>>> > >
>>>> > > Please vote on releasing the following candidate as Apache Spark
>>>> version 2.4.0.
>>>> > >
>>>> > > The vote is open until October 1 PST and passes if a majority +1
>>>> PMC votes are cast, with
>>>> > > a minimum of 3 +1 votes.
>>>> > >
>>>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>>>> > > [ ] -1 Do not release this package because ...
>>>> > >
>>>> > > To learn more about Apache Spark, please see
>>>> http://spark.apache.org/
>>>> > >
>>>> > > The tag to be voted on is v2.4.0-rc3 (commit
>>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>>>> > >
>>>> > > The release files, including signatures, digests, etc. can be found
>>>> at:
>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>> > >
>>>> > > Signatures used for Spark RCs can be found in this file:
>>>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>> > >
>>>> > > The staging repository for this release can be found at:
>>>> > >
>>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>> > >
>>>> > > The documentation corresponding to this release can be found at:
>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>> > >
>>>> > > The list of bug fixes going into 2.4.0 can be found at the
>>>> following URL:
>>>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>> > >
>>>> > > FAQ
>>>> > >
>>>> > > =========================
>>>> > > How can I help test this release?
>>>> > > =========================
>>>> > >
>>>> > > If you are a Spark user, you can help us test this release by taking
>>>> > > an existing Spark workload and running on this release candidate,
>>>> then
>>>> > > reporting any regressions.
>>>> > >
>>>> > > If you're working in PySpark you can set up a virtual env and
>>>> install
>>>> > > the current RC and see if anything important breaks, in the
>>>> Java/Scala
>>>> > > you can add the staging repository to your projects resolvers and
>>>> test
>>>> > > with the RC (make sure to clean up the artifact cache before/after
>>>> so
>>>> > > you don't end up building with a out of date RC going forward).
>>>> > >
>>>> > > ===========================================
>>>> > > What should happen to JIRA tickets still targeting 2.4.0?
>>>> > > ===========================================
>>>> > >
>>>> > > The current list of open tickets targeted at 2.4.0 can be found at:
>>>> > > https://issues.apache.org/jira/projects/SPARK and search for
>>>> "Target Version/s" = 2.4.0
>>>> > >
>>>> > > Committers should look at those and triage. Extremely important bug
>>>> > > fixes, documentation, and API tweaks that impact compatibility
>>>> should
>>>> > > be worked on immediately. Everything else please retarget to an
>>>> > > appropriate release.
>>>> > >
>>>> > > ==================
>>>> > > But my bug isn't fixed?
>>>> > > ==================
>>>> > >
>>>> > > In order to make timely releases, we will typically not hold the
>>>> > > release unless the bug in question is a regression from the previous
>>>> > > release. That being said, if there is something which is a
>>>> regression
>>>> > > that has not been correctly targeted please ping me or a committer
>>>> to
>>>> > > help target the issue.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Dongjoon Hyun <do...@gmail.com>.
Hi, Holden.
Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.
Bests,
Dongjoon.
On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <ho...@pigscanfly.ca> wrote:
> Following up I just wanted to make sure this new blocker that Dongjoon
> designated is surfaced -
> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409&jql=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
>
> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <ga...@gmail.com> wrote:
>
>> -1. We have two correctness bugs:
>> https://issues.apache.org/jira/browse/SPARK-25714 and
>> https://issues.apache.org/jira/browse/SPARK-25708.
>>
>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>>
>> Xiao
>>
>>
>> Sean Owen <sr...@apache.org> 于2018年10月11日周四 上午9:04写道：
>>
>>> This is a legitimate question about the behavior of ScalaUDF after the
>>> change to support 2.12:
>>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>>> Not quite a blocker I think, but a potential gotcha we definitely need
>>> to highlight in release notes. There may be an argument for changing
>>> ScalaUDF again before the release. Have a look, anyone familiar with
>>> catalyst.
>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
>>> >
>>> > +1. I tested the source build against Scala 2.12 and common build
>>> > profiles. License and sigs look OK.
>>> >
>>> > No blockers; one critical:
>>> >
>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>>> >
>>> > I think this one is "won't fix" though? not trying to restore the
>>> behavior?
>>> >
>>> > Other items open for 2.4.0:
>>> >
>>> > SPARK-25347 Document image data source in doc site
>>> > SPARK-25584 Document libsvm data source in doc site
>>> > SPARK-25179 Document the features that require Pyarrow 0.10
>>> > SPARK-25507 Update documents for the new features in 2.4 release
>>> > SPARK-25346 Document Spark builtin data sources
>>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>>> > SPARK-23197 Flaky test:
>>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>>> > SPARK-22809 pyspark is sensitive to imports with dots
>>> > SPARK-21030 extend hint syntax to support any expression for Python
>>> and R
>>> >
>>> > Anyone know enough to close or retarget them? they don't look critical
>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
>>> > features that require Pyarrow 0.10" however sounds like it could have
>>> > been important for 2.4? if not a blocker.
>>> >
>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>>> > there is some ongoing discussion there.
>>> >
>>> > I am evaluating
>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642 right
>>> > now.
>>> >
>>> >
>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com>
>>> wrote:
>>> > >
>>> > > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.0.
>>> > >
>>> > > The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> > > a minimum of 3 +1 votes.
>>> > >
>>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>>> > > [ ] -1 Do not release this package because ...
>>> > >
>>> > > To learn more about Apache Spark, please see
>>> http://spark.apache.org/
>>> > >
>>> > > The tag to be voted on is v2.4.0-rc3 (commit
>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>>> > >
>>> > > The release files, including signatures, digests, etc. can be found
>>> at:
>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>> > >
>>> > > Signatures used for Spark RCs can be found in this file:
>>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> > >
>>> > > The staging repository for this release can be found at:
>>> > >
>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>> > >
>>> > > The documentation corresponding to this release can be found at:
>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>> > >
>>> > > The list of bug fixes going into 2.4.0 can be found at the following
>>> URL:
>>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>> > >
>>> > > FAQ
>>> > >
>>> > > =========================
>>> > > How can I help test this release?
>>> > > =========================
>>> > >
>>> > > If you are a Spark user, you can help us test this release by taking
>>> > > an existing Spark workload and running on this release candidate,
>>> then
>>> > > reporting any regressions.
>>> > >
>>> > > If you're working in PySpark you can set up a virtual env and install
>>> > > the current RC and see if anything important breaks, in the
>>> Java/Scala
>>> > > you can add the staging repository to your projects resolvers and
>>> test
>>> > > with the RC (make sure to clean up the artifact cache before/after so
>>> > > you don't end up building with a out of date RC going forward).
>>> > >
>>> > > ===========================================
>>> > > What should happen to JIRA tickets still targeting 2.4.0?
>>> > > ===========================================
>>> > >
>>> > > The current list of open tickets targeted at 2.4.0 can be found at:
>>> > > https://issues.apache.org/jira/projects/SPARK and search for
>>> "Target Version/s" = 2.4.0
>>> > >
>>> > > Committers should look at those and triage. Extremely important bug
>>> > > fixes, documentation, and API tweaks that impact compatibility should
>>> > > be worked on immediately. Everything else please retarget to an
>>> > > appropriate release.
>>> > >
>>> > > ==================
>>> > > But my bug isn't fixed?
>>> > > ==================
>>> > >
>>> > > In order to make timely releases, we will typically not hold the
>>> > > release unless the bug in question is a regression from the previous
>>> > > release. That being said, if there is something which is a regression
>>> > > that has not been correctly targeted please ping me or a committer to
>>> > > help target the issue.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Holden Karau <ho...@pigscanfly.ca>.
Following up I just wanted to make sure this new blocker that Dongjoon
designated is surfaced -
https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409&jql=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker
On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <ga...@gmail.com> wrote:
> -1. We have two correctness bugs:
> https://issues.apache.org/jira/browse/SPARK-25714 and
> https://issues.apache.org/jira/browse/SPARK-25708.
>
> Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
>
> Xiao
>
>
> Sean Owen <sr...@apache.org> 于2018年10月11日周四 上午9:04写道：
>
>> This is a legitimate question about the behavior of ScalaUDF after the
>> change to support 2.12:
>> https://github.com/apache/spark/pull/22259#discussion_r224295469
>> Not quite a blocker I think, but a potential gotcha we definitely need
>> to highlight in release notes. There may be an argument for changing
>> ScalaUDF again before the release. Have a look, anyone familiar with
>> catalyst.
>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
>> >
>> > +1. I tested the source build against Scala 2.12 and common build
>> > profiles. License and sigs look OK.
>> >
>> > No blockers; one critical:
>> >
>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>> >
>> > I think this one is "won't fix" though? not trying to restore the
>> behavior?
>> >
>> > Other items open for 2.4.0:
>> >
>> > SPARK-25347 Document image data source in doc site
>> > SPARK-25584 Document libsvm data source in doc site
>> > SPARK-25179 Document the features that require Pyarrow 0.10
>> > SPARK-25507 Update documents for the new features in 2.4 release
>> > SPARK-25346 Document Spark builtin data sources
>> > SPARK-24464 Unit tests for MLlib's Instrumentation
>> > SPARK-23197 Flaky test:
>> spark.streaming.ReceiverSuite."receiver_life_cycle"
>> > SPARK-22809 pyspark is sensitive to imports with dots
>> > SPARK-21030 extend hint syntax to support any expression for Python and
>> R
>> >
>> > Anyone know enough to close or retarget them? they don't look critical
>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
>> > features that require Pyarrow 0.10" however sounds like it could have
>> > been important for 2.4? if not a blocker.
>> >
>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
>> > there is some ongoing discussion there.
>> >
>> > I am evaluating
>> > https://github.com/apache/spark/pull/22259#discussion_r224252642 right
>> > now.
>> >
>> >
>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com>
>> wrote:
>> > >
>> > > Please vote on releasing the following candidate as Apache Spark
>> version 2.4.0.
>> > >
>> > > The vote is open until October 1 PST and passes if a majority +1 PMC
>> votes are cast, with
>> > > a minimum of 3 +1 votes.
>> > >
>> > > [ ] +1 Release this package as Apache Spark 2.4.0
>> > > [ ] -1 Do not release this package because ...
>> > >
>> > > To learn more about Apache Spark, please see http://spark.apache.org/
>> > >
>> > > The tag to be voted on is v2.4.0-rc3 (commit
>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> > > https://github.com/apache/spark/tree/v2.4.0-rc3
>> > >
>> > > The release files, including signatures, digests, etc. can be found
>> at:
>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>> > >
>> > > Signatures used for Spark RCs can be found in this file:
>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> > >
>> > > The staging repository for this release can be found at:
>> > >
>> https://repository.apache.org/content/repositories/orgapachespark-1289
>> > >
>> > > The documentation corresponding to this release can be found at:
>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>> > >
>> > > The list of bug fixes going into 2.4.0 can be found at the following
>> URL:
>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
>> > >
>> > > FAQ
>> > >
>> > > =========================
>> > > How can I help test this release?
>> > > =========================
>> > >
>> > > If you are a Spark user, you can help us test this release by taking
>> > > an existing Spark workload and running on this release candidate, then
>> > > reporting any regressions.
>> > >
>> > > If you're working in PySpark you can set up a virtual env and install
>> > > the current RC and see if anything important breaks, in the Java/Scala
>> > > you can add the staging repository to your projects resolvers and test
>> > > with the RC (make sure to clean up the artifact cache before/after so
>> > > you don't end up building with a out of date RC going forward).
>> > >
>> > > ===========================================
>> > > What should happen to JIRA tickets still targeting 2.4.0?
>> > > ===========================================
>> > >
>> > > The current list of open tickets targeted at 2.4.0 can be found at:
>> > > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.0
>> > >
>> > > Committers should look at those and triage. Extremely important bug
>> > > fixes, documentation, and API tweaks that impact compatibility should
>> > > be worked on immediately. Everything else please retarget to an
>> > > appropriate release.
>> > >
>> > > ==================
>> > > But my bug isn't fixed?
>> > > ==================
>> > >
>> > > In order to make timely releases, we will typically not hold the
>> > > release unless the bug in question is a regression from the previous
>> > > release. That being said, if there is something which is a regression
>> > > that has not been correctly targeted please ping me or a committer to
>> > > help target the issue.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Xiao Li <ga...@gmail.com>.
-1. We have two correctness bugs:
https://issues.apache.org/jira/browse/SPARK-25714 and
https://issues.apache.org/jira/browse/SPARK-25708.
Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
Xiao
Sean Owen <sr...@apache.org> 于2018年10月11日周四 上午9:04写道：
> This is a legitimate question about the behavior of ScalaUDF after the
> change to support 2.12:
> https://github.com/apache/spark/pull/22259#discussion_r224295469
> Not quite a blocker I think, but a potential gotcha we definitely need
> to highlight in release notes. There may be an argument for changing
> ScalaUDF again before the release. Have a look, anyone familiar with
> catalyst.
> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
> >
> > +1. I tested the source build against Scala 2.12 and common build
> > profiles. License and sigs look OK.
> >
> > No blockers; one critical:
> >
> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
> >
> > I think this one is "won't fix" though? not trying to restore the
> behavior?
> >
> > Other items open for 2.4.0:
> >
> > SPARK-25347 Document image data source in doc site
> > SPARK-25584 Document libsvm data source in doc site
> > SPARK-25179 Document the features that require Pyarrow 0.10
> > SPARK-25507 Update documents for the new features in 2.4 release
> > SPARK-25346 Document Spark builtin data sources
> > SPARK-24464 Unit tests for MLlib's Instrumentation
> > SPARK-23197 Flaky test:
> spark.streaming.ReceiverSuite."receiver_life_cycle"
> > SPARK-22809 pyspark is sensitive to imports with dots
> > SPARK-21030 extend hint syntax to support any expression for Python and R
> >
> > Anyone know enough to close or retarget them? they don't look critical
> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> > features that require Pyarrow 0.10" however sounds like it could have
> > been important for 2.4? if not a blocker.
> >
> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> > there is some ongoing discussion there.
> >
> > I am evaluating
> > https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> > now.
> >
> >
> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com> wrote:
> > >
> > > Please vote on releasing the following candidate as Apache Spark
> version 2.4.0.
> > >
> > > The vote is open until October 1 PST and passes if a majority +1 PMC
> votes are cast, with
> > > a minimum of 3 +1 votes.
> > >
> > > [ ] +1 Release this package as Apache Spark 2.4.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > To learn more about Apache Spark, please see http://spark.apache.org/
> > >
> > > The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > > https://github.com/apache/spark/tree/v2.4.0-rc3
> > >
> > > The release files, including signatures, digests, etc. can be found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> > >
> > > Signatures used for Spark RCs can be found in this file:
> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
> > >
> > > The staging repository for this release can be found at:
> > > https://repository.apache.org/content/repositories/orgapachespark-1289
> > >
> > > The documentation corresponding to this release can be found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> > >
> > > The list of bug fixes going into 2.4.0 can be found at the following
> URL:
> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> > >
> > > FAQ
> > >
> > > =========================
> > > How can I help test this release?
> > > =========================
> > >
> > > If you are a Spark user, you can help us test this release by taking
> > > an existing Spark workload and running on this release candidate, then
> > > reporting any regressions.
> > >
> > > If you're working in PySpark you can set up a virtual env and install
> > > the current RC and see if anything important breaks, in the Java/Scala
> > > you can add the staging repository to your projects resolvers and test
> > > with the RC (make sure to clean up the artifact cache before/after so
> > > you don't end up building with a out of date RC going forward).
> > >
> > > ===========================================
> > > What should happen to JIRA tickets still targeting 2.4.0?
> > > ===========================================
> > >
> > > The current list of open tickets targeted at 2.4.0 can be found at:
> > > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
> > >
> > > Committers should look at those and triage. Extremely important bug
> > > fixes, documentation, and API tweaks that impact compatibility should
> > > be worked on immediately. Everything else please retarget to an
> > > appropriate release.
> > >
> > > ==================
> > > But my bug isn't fixed?
> > > ==================
> > >
> > > In order to make timely releases, we will typically not hold the
> > > release unless the bug in question is a regression from the previous
> > > release. That being said, if there is something which is a regression
> > > that has not been correctly targeted please ping me or a committer to
> > > help target the issue.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Sean Owen <sr...@apache.org>.
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sr...@apache.org> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Fwd: [VOTE] SPARK 2.4.0 (RC3)
Posted by Wenchen Fan <cl...@gmail.com>.
Forgot to cc dev-list
---------- Forwarded message ---------
From: Wenchen Fan <cl...@gmail.com>
Date: Thu, Oct 11, 2018 at 10:14 AM
Subject: Re: [VOTE] SPARK 2.4.0 (RC3)
To: <he...@gmail.com>
Cc: Sean Owen <sr...@apache.org>
Ah sorry guys, I just copy-paste the voting email from the last RC and
forgot to update the date :P
The voting should be open until October 13 PST.
According to the discussion in the previous RC, I'm resolving SPARK-25378
as won't fix. It's OK to wait one or 2 weeks for the tensorflow release.
SPARK-25150 is a long-standing and known issue I believe, DataFrame join
API may have confusing behavior for indirect self-join, and is relatively
hard to fix, if breaking change is not allowed. I've seen many tickets
complaining about it and we should definitely fix it in 3.0, which accepts
necessary breaking changes.
SPARK-25588 does look like a potential issue, but there is not much we can
do if this problem is not reproducible.
On Thu, Oct 11, 2018 at 7:28 AM Michael Heuer <he...@gmail.com> wrote:
> Hello Sean, Wenchen
>
> I could use triage on
>
> https://issues.apache.org/jira/browse/SPARK-25588
>
> I’ve struggled reporting Parquet+Avro dependency issues against Spark in
> the past, can’t seem to get any notice.
>
> michael
>
>
> On Oct 10, 2018, at 5:00 PM, Sean Owen <sr...@apache.org> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Sean Owen <sr...@apache.org>.
+1. I tested the source build against Scala 2.12 and common build
profiles. License and sigs look OK.
No blockers; one critical:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
I think this one is "won't fix" though? not trying to restore the behavior?
Other items open for 2.4.0:
SPARK-25347 Document image data source in doc site
SPARK-25584 Document libsvm data source in doc site
SPARK-25179 Document the features that require Pyarrow 0.10
SPARK-25507 Update documents for the new features in 2.4 release
SPARK-25346 Document Spark builtin data sources
SPARK-24464 Unit tests for MLlib's Instrumentation
SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
SPARK-22809 pyspark is sensitive to imports with dots
SPARK-21030 extend hint syntax to support any expression for Python and R
Anyone know enough to close or retarget them? they don't look critical
for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
features that require Pyarrow 0.10" however sounds like it could have
been important for 2.4? if not a blocker.
PS I don't think that SPARK-25150 is an issue; see JIRA. At least
there is some ongoing discussion there.
I am evaluating
https://github.com/apache/spark/pull/22259#discussion_r224252642 right
now.
On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Nicholas Chammas <ni...@gmail.com>.
FYI I believe we have an open correctness issue here:
https://issues.apache.org/jira/browse/SPARK-25150
However, it needs review by another person to confirm whether it is indeed
a correctness issue (and whether it still impacts this latest RC).
Nick
2018년 10월 10일 (수) 오후 3:14, Jean Georges Perrin <jg...@jgp.net>님이 작성:
> Awesome - thanks Dongjoon!
>
>
> On Oct 10, 2018, at 2:36 PM, Dongjoon Hyun <do...@gmail.com>
> wrote:
>
> For now, you can see generated release notes. Official one will be posted
> on the website when the official 2.4.0 is out.
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12342385
>
> Bests,
> Dongjoon.
>
>
> On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin <jg...@jgp.net> wrote:
>
>> Hi,
>>
>> Sorry if it's stupid question, but where can I find the release notes of
>> 2.4.0?
>>
>> jg
>>
>> On Oct 10, 2018, at 2:00 PM, Imran Rashid <ir...@cloudera.com.INVALID>
>> wrote:
>>
>> Sorry I had messed up my testing earlier, so I only just discovered
>> https://issues.apache.org/jira/browse/SPARK-25704
>>
>> I dont' think this is a release blocker, because its not a regression and
>> there is a workaround, just fyi.
>>
>> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.4.0.
>>>
>>> The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.4.0
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.4.0-rc3 (commit
>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>
>>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>
>>> FAQ
>>>
>>> =========================
>>> How can I help test this release?
>>> =========================
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===========================================
>>> What should happen to JIRA tickets still targeting 2.4.0?
>>> ===========================================
>>>
>>> The current list of open tickets targeted at 2.4.0 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==================
>>> But my bug isn't fixed?
>>> ==================
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>
>>
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Jean Georges Perrin <jg...@jgp.net>.
Awesome - thanks Dongjoon!
> On Oct 10, 2018, at 2:36 PM, Dongjoon Hyun <do...@gmail.com> wrote:
>
> For now, you can see generated release notes. Official one will be posted on the website when the official 2.4.0 is out.
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12342385 <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12342385>
>
> Bests,
> Dongjoon.
>
>
> On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
> Hi,
>
> Sorry if it's stupid question, but where can I find the release notes of 2.4.0?
>
> jg
>
>> On Oct 10, 2018, at 2:00 PM, Imran Rashid <irashid@cloudera.com.INVALID <ma...@cloudera.com.INVALID>> wrote:
>>
>> Sorry I had messed up my testing earlier, so I only just discovered https://issues.apache.org/jira/browse/SPARK-25704 <https://issues.apache.org/jira/browse/SPARK-25704>
>>
>> I dont' think this is a release blocker, because its not a regression and there is a workaround, just fyi.
>>
>> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <cloud0fan@gmail.com <ma...@gmail.com>> wrote:
>> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/ <http://spark.apache.org/>
>>
>> The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> https://github.com/apache/spark/tree/v2.4.0-rc3 <https://github.com/apache/spark/tree/v2.4.0-rc3>
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/ <https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/>
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1289 <https://repository.apache.org/content/repositories/orgapachespark-1289>
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/ <https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/>
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12342385 <https://issues.apache.org/jira/projects/SPARK/versions/12342385>
>>
>> FAQ
>>
>> =========================
>> How can I help test this release?
>> =========================
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===========================================
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===========================================
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK <https://issues.apache.org/jira/projects/SPARK> and search for "Target Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==================
>> But my bug isn't fixed?
>> ==================
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Dongjoon Hyun <do...@gmail.com>.
For now, you can see generated release notes. Official one will be posted
on the website when the official 2.4.0 is out.
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12342385
Bests,
Dongjoon.
On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin <jg...@jgp.net> wrote:
> Hi,
>
> Sorry if it's stupid question, but where can I find the release notes of
> 2.4.0?
>
> jg
>
> On Oct 10, 2018, at 2:00 PM, Imran Rashid <ir...@cloudera.com.INVALID>
> wrote:
>
> Sorry I had messed up my testing earlier, so I only just discovered
> https://issues.apache.org/jira/browse/SPARK-25704
>
> I dont' think this is a release blocker, because its not a regression and
> there is a workaround, just fyi.
>
> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC
>> votes are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.0-rc3 (commit
>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>
>> FAQ
>>
>> =========================
>> How can I help test this release?
>> =========================
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===========================================
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===========================================
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==================
>> But my bug isn't fixed?
>> ==================
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Jean Georges Perrin <jg...@jgp.net>.
Hi,
Sorry if it's stupid question, but where can I find the release notes of 2.4.0?
jg
> On Oct 10, 2018, at 2:00 PM, Imran Rashid <irashid@cloudera.com.INVALID <ma...@cloudera.com.INVALID>> wrote:
>
> Sorry I had messed up my testing earlier, so I only just discovered https://issues.apache.org/jira/browse/SPARK-25704 <https://issues.apache.org/jira/browse/SPARK-25704>
>
> I dont' think this is a release blocker, because its not a regression and there is a workaround, just fyi.
>
> On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <cloud0fan@gmail.com <ma...@gmail.com>> wrote:
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/ <http://spark.apache.org/>
>
> The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3 <https://github.com/apache/spark/tree/v2.4.0-rc3>
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/ <https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/>
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS <https://dist.apache.org/repos/dist/dev/spark/KEYS>
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289 <https://repository.apache.org/content/repositories/orgapachespark-1289>
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/ <https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/>
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385 <https://issues.apache.org/jira/projects/SPARK/versions/12342385>
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK <https://issues.apache.org/jira/projects/SPARK> and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Imran Rashid <ir...@cloudera.com.INVALID>.
Sorry I had messed up my testing earlier, so I only just discovered
https://issues.apache.org/jira/browse/SPARK-25704
I dont' think this is a release blocker, because its not a regression and
there is a workaround, just fyi.
On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <cl...@gmail.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Hyukjin Kwon <gu...@gmail.com>.
So, which date is it?
2018년 10월 11일 (목) 오전 1:48, Garlapati, Suryanarayana (Nokia - IN/Bangalore) <
suryanarayana.garlapati@nokia.com>님이 작성:
> Might be you need to change the date(Oct 1 has already passed).
>
>
>
> >> The vote is open until October 1 PST and passes if a majority +1 PMC
> votes are cast, with
>
> >> a minimum of 3 +1 votes.
>
>
>
> Regards
>
> Surya
>
>
>
> *From:* Wenchen Fan <cl...@gmail.com>
> *Sent:* Wednesday, October 10, 2018 10:20 PM
> *To:* Spark dev list <de...@spark.apache.org>
> *Subject:* Re: [VOTE] SPARK 2.4.0 (RC3)
>
>
>
> I'm adding my own +1, since there are no known blocker issues. The
> correctness issue has been fixed, the streaming Java API problem has been
> resolved, and we have upgraded to Scala 2.12.7.
>
>
>
> On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan <cl...@gmail.com> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
>
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
>
> a minimum of 3 +1 votes.
>
>
>
> [ ] +1 Release this package as Apache Spark 2.4.0
>
> [ ] -1 Do not release this package because ...
>
>
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
>
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
>
>
> The release files, including signatures, digests, etc. can be found at:
>
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
>
>
> Signatures used for Spark RCs can be found in this file:
>
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
>
>
> The staging repository for this release can be found at:
>
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
>
>
> The documentation corresponding to this release can be found at:
>
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
>
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
>
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
>
>
> FAQ
>
>
>
> =========================
>
> How can I help test this release?
>
> =========================
>
>
>
> If you are a Spark user, you can help us test this release by taking
>
> an existing Spark workload and running on this release candidate, then
>
> reporting any regressions.
>
>
>
> If you're working in PySpark you can set up a virtual env and install
>
> the current RC and see if anything important breaks, in the Java/Scala
>
> you can add the staging repository to your projects resolvers and test
>
> with the RC (make sure to clean up the artifact cache before/after so
>
> you don't end up building with a out of date RC going forward).
>
>
>
> ===========================================
>
> What should happen to JIRA tickets still targeting 2.4.0?
>
> ===========================================
>
>
>
> The current list of open tickets targeted at 2.4.0 can be found at:
>
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
>
>
> Committers should look at those and triage. Extremely important bug
>
> fixes, documentation, and API tweaks that impact compatibility should
>
> be worked on immediately. Everything else please retarget to an
>
> appropriate release.
>
>
>
> ==================
>
> But my bug isn't fixed?
>
> ==================
>
>
>
> In order to make timely releases, we will typically not hold the
>
> release unless the bug in question is a regression from the previous
>
> release. That being said, if there is something which is a regression
>
> that has not been correctly targeted please ping me or a committer to
>
> help target the issue.
>
>
RE: [VOTE] SPARK 2.4.0 (RC3)
Posted by "Garlapati, Suryanarayana (Nokia - IN/Bangalore)" <su...@nokia.com>.
Might be you need to change the date(Oct 1 has already passed).
>> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
>> a minimum of 3 +1 votes.
Regards
Surya
From: Wenchen Fan <cl...@gmail.com>
Sent: Wednesday, October 10, 2018 10:20 PM
To: Spark dev list <de...@spark.apache.org>
Subject: Re: [VOTE] SPARK 2.4.0 (RC3)
I'm adding my own +1, since there are no known blocker issues. The correctness issue has been fixed, the streaming Java API problem has been resolved, and we have upgraded to Scala 2.12.7.
On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan <cl...@gmail.com>> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.
The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.
[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...
To learn more about Apache Spark, please see http://spark.apache.org/
The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
https://github.com/apache/spark/tree/v2.4.0-rc3
The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS
The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1289
The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
The list of bug fixes going into 2.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12342385
FAQ
=========================
How can I help test this release?
=========================
If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.
If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).
===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================
The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.
==================
But my bug isn't fixed?
==================
In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Wenchen Fan <cl...@gmail.com>.
I'm adding my own +1, since there are no known blocker issues. The
correctness issue has been fixed, the streaming Java API problem has been
resolved, and we have upgraded to Scala 2.12.7.
On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan <cl...@gmail.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you, Wenchen. I made the minor document-only change PR.
https://github.com/apache/spark/pull/22781
Bests,
Dongjoon.
On Fri, Oct 19, 2018 at 6:07 PM Wenchen Fan <cl...@gmail.com> wrote:
> AFAIK we haven't tested Java 9+ yet, so I'm ok to change it.
>
> Hi Dongjoon can you make a PR for it? We can merge it very soon if we
> decide to do it.
>
> Thanks,
> Wenchen
>
> On Sat, Oct 20, 2018 at 5:27 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> From the document, should we be more specific with 'Java 8' instead of
>> 'Java 8+' because we don't build (or test) in the community with Java 9 ~
>> 11.
>>
>>
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/building-spark.html
>> > Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.4.0.
>>>
>>> The vote is open until October 1 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.4.0
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.4.0-rc3 (commit
>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>>
>>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>>
>>> FAQ
>>>
>>> =========================
>>> How can I help test this release?
>>> =========================
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===========================================
>>> What should happen to JIRA tickets still targeting 2.4.0?
>>> ===========================================
>>>
>>> The current list of open tickets targeted at 2.4.0 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 2.4.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==================
>>> But my bug isn't fixed?
>>> ==================
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Wenchen Fan <cl...@gmail.com>.
AFAIK we haven't tested Java 9+ yet, so I'm ok to change it.
Hi Dongjoon can you make a PR for it? We can merge it very soon if we
decide to do it.
Thanks,
Wenchen
On Sat, Oct 20, 2018 at 5:27 AM Dongjoon Hyun <do...@gmail.com>
wrote:
> From the document, should we be more specific with 'Java 8' instead of
> 'Java 8+' because we don't build (or test) in the community with Java 9 ~
> 11.
>
>
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/building-spark.html
> > Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+
>
> Bests,
> Dongjoon.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC
>> votes are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.0-rc3 (commit
>> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
>> https://github.com/apache/spark/tree/v2.4.0-rc3
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1289
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>>
>> FAQ
>>
>> =========================
>> How can I help test this release?
>> =========================
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===========================================
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===========================================
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==================
>> But my bug isn't fixed?
>> ==================
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>
Re: [VOTE] SPARK 2.4.0 (RC3)
Posted by Dongjoon Hyun <do...@gmail.com>.
From the document, should we be more specific with 'Java 8' instead of
'Java 8+' because we don't build (or test) in the community with Java 9 ~
11.
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/building-spark.html
> Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+
Bests,
Dongjoon.
On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cl...@gmail.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit
> 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>