You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by DB Tsai <d_...@apple.com.INVALID> on 2019/03/08 00:15:06 UTC
[VOTE] Release Apache Spark 2.4.1 (RC6)
Please vote on releasing the following candidate as Apache Spark version 2.4.1.
The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.
[ ] +1 Release this package as Apache Spark 2.4.1
[ ] -1 Do not release this package because ...
To learn more about Apache Spark, please see http://spark.apache.org/
The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
https://github.com/apache/spark/tree/v2.4.1-rc6
The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS
The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1308/
The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
The list of bug fixes going into 2.4.1 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
FAQ
=========================
How can I help test this release?
=========================
If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.
If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).
===========================================
What should happen to JIRA tickets still targeting 2.4.1?
===========================================
The current list of open tickets targeted at 2.4.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.
==================
But my bug isn't fixed?
==================
In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Marcelo Vanzin <va...@cloudera.com.INVALID>.
I'd be more comfortable with an rc7. Either that or manually fix the
branch with a force push, but that's a bit risky, it's easy to mess up
force pushes (if we can even do that?).
It's very possible that there is a bug in the script; IIRC it should
create the commits in the right branch when you generate the rc the
first time. Perhaps you missed some error in the command line in that
invocation (tag was created but commits not pushed to the branch, for
example).
On Fri, Mar 8, 2019 at 11:39 AM DB Tsai <d_...@apple.com> wrote:
>
> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>
> Should we create a new rc7?
>
> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>
> > On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <va...@cloudera.com.INVALID> wrote:
> >
> > I personally find it a little weird to not have the commit in branch-2.4.
> >
> > Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
> > (e.g. accidentally) then you lose the reference to that commit, and
> > then the exact commit from which the rc was generated is lost.
> >
> > On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
> >>
> >> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
> >>
> >> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
> >>
> >> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
> >>>
> >>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
> >>>
> >>> Best Regards,
> >>>
> >>> Misi
> >>>
> >>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P, 1:15):
> >>>>
> >>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
> >>>>
> >>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
> >>>> a minimum of 3 +1 votes.
> >>>>
> >>>> [ ] +1 Release this package as Apache Spark 2.4.1
> >>>> [ ] -1 Do not release this package because ...
> >>>>
> >>>> To learn more about Apache Spark, please see http://spark.apache.org/
> >>>>
> >>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
> >>>> https://github.com/apache/spark/tree/v2.4.1-rc6
> >>>>
> >>>> The release files, including signatures, digests, etc. can be found at:
> >>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
> >>>>
> >>>> Signatures used for Spark RCs can be found in this file:
> >>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
> >>>>
> >>>> The staging repository for this release can be found at:
> >>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
> >>>>
> >>>> The documentation corresponding to this release can be found at:
> >>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
> >>>>
> >>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
> >>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
> >>>>
> >>>> FAQ
> >>>>
> >>>> =========================
> >>>> How can I help test this release?
> >>>> =========================
> >>>>
> >>>> If you are a Spark user, you can help us test this release by taking
> >>>> an existing Spark workload and running on this release candidate, then
> >>>> reporting any regressions.
> >>>>
> >>>> If you're working in PySpark you can set up a virtual env and install
> >>>> the current RC and see if anything important breaks, in the Java/Scala
> >>>> you can add the staging repository to your projects resolvers and test
> >>>> with the RC (make sure to clean up the artifact cache before/after so
> >>>> you don't end up building with a out of date RC going forward).
> >>>>
> >>>> ===========================================
> >>>> What should happen to JIRA tickets still targeting 2.4.1?
> >>>> ===========================================
> >>>>
> >>>> The current list of open tickets targeted at 2.4.1 can be found at:
> >>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
> >>>>
> >>>> Committers should look at those and triage. Extremely important bug
> >>>> fixes, documentation, and API tweaks that impact compatibility should
> >>>> be worked on immediately. Everything else please retarget to an
> >>>> appropriate release.
> >>>>
> >>>> ==================
> >>>> But my bug isn't fixed?
> >>>> ==================
> >>>>
> >>>> In order to make timely releases, we will typically not hold the
> >>>> release unless the bug in question is a regression from the previous
> >>>> release. That being said, if there is something which is a regression
> >>>> that has not been correctly targeted please ping me or a committer to
> >>>> help target the issue.
> >>>>
> >>>> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>>
> >
> >
> > --
> > Marcelo
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Michael Heuer <he...@gmail.com>.
I'm not saying that this issue should be a blocker for 2.4.1, rather I'm looking for help moving things along. I'm not a committer in any of the Spark, Parquet, or Avro projects.
> On Mar 10, 2019, at 8:53 PM, Sean Owen <sr...@gmail.com> wrote:
>
> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:
>
> - this is a Parquet-Avro version conflict thing
> - a downstream app wants different versions of Parquet and Avro than
> Spark uses, which triggers it
Prior to 2.4.0, Spark depended on versions of Parquet and Avro that did not work with each other. In fact, a different version of Avro had to be used in Spark's test scope to prevent runtime errors.
As a workaround, we had to override Parquet to 1.8.2 (later 1.8.3) but pin parquet-avro to 1.8.1.
> - it doesn't work in 2.4.0
In 2.4.0 we're no longer able to pin parquet-avro to 1.8.1, so our workaround is broken. Using the Spark 2.4.0 versions of Parquet and Avro uncovers this new error.
>
> It's not a regression from 2.4.0, which is the immediate question.
> There isn't even a Parquet fix available.
I believe https://github.com/apache/parquet-mr/pull/560/files <https://github.com/apache/parquet-mr/pull/560/files> is a fix, but I haven't made all the necessary snapshot builds to test. I was waiting for a valid Spark 2.4.1 RC to try.
> But I'm not even seeing why this is excuse-making?
>
> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com> wrote:
>>
>> Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Takeshi Yamamuro <li...@gmail.com>.
Hi,
It might be worth including the fix blow in the next candidate?:
https://github.com/apache/spark/pull/24085
On Mon, Mar 18, 2019 at 4:54 PM DB Tsai <db...@dbtsai.com.invalid> wrote:
> Rc6 was failed due to the critical bug Xiao mentioned. Rc8 was cut to
> include those fixes, but I ran into some nexus infra issues when I tried to
> upload the jars into Apache repo.
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/INFRA-18018
>
> The infra team reordered the key servers to prevent the timeout, and
> rebooted the server. I am running the build again to see if I can
> successfully upload Spark jars this time. Thanks!
>
> On Sun, Mar 17, 2019 at 2:32 PM Koert Kuipers <ko...@tresata.com> wrote:
>
>> hi all,
>> i am having a bit trouble following the RCs this time. is RC6 still the
>> one being voted on? i see on github v2.4.1-rc7 and v2.4.1-rc8 but no email
>> threads.
>> thanks!
>> best, koert
>>
>> On Mon, Mar 11, 2019 at 1:32 AM Xiao Li <li...@databricks.com> wrote:
>>
>>> ++ Ryan Blue. What is your opinion regarding this Parquet issue?
>>>
>>> Thanks,
>>>
>>> Xiao
>>>
>>> On Sun, Mar 10, 2019 at 9:58 PM Wenchen Fan <cl...@gmail.com> wrote:
>>>
>>>> Which version of Parquet has this bug? Maybe we can downgrade it.
>>>>
>>>> On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <ma...@clearstorydata.com>
>>>> wrote:
>>>>
>>>>> It worked in 2.3. We broke it with 2.4.0 and were informed of that
>>>>> regression late in the 2.4.0 release process. Since we didn't fix it before
>>>>> the 2.4.0 release, it should have been noted as a known issue. To now claim
>>>>> that there is no regression from 2.4.0 is a circular argument denying the
>>>>> existence of a known regression from 2.3.
>>>>>
>>>>> On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>>>
>>>>>> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading
>>>>>> that:
>>>>>>
>>>>>> - this is a Parquet-Avro version conflict thing
>>>>>> - a downstream app wants different versions of Parquet and Avro than
>>>>>> Spark uses, which triggers it
>>>>>> - it doesn't work in 2.4.0
>>>>>>
>>>>>> It's not a regression from 2.4.0, which is the immediate question.
>>>>>> There isn't even a Parquet fix available.
>>>>>> But I'm not even seeing why this is excuse-making?
>>>>>>
>>>>>> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Now wait... we created a regression in 2.4.0. Arguably, we should
>>>>>> have blocked that release until we had a fix; but the issue came up late in
>>>>>> the release process and it looks to me like there wasn't an adequate fix
>>>>>> immediately available, so we did something bad and released 2.4.0 with a
>>>>>> known regression. Saying that there is now no regression from 2.4 is
>>>>>> tautological and no excuse for not taking in a fix -- and it looks like
>>>>>> that fix has been waiting for months.
>>>>>>
>>>>>
>>>
>>> --
>>> [image:
>>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>>
>> --
> - DB Sent from my iPhone
>
--
---
Takeshi Yamamuro
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by shane knapp <sk...@berkeley.edu>.
i'd like to make sure https://github.com/apache/spark/pull/24137 is
included in the 2.4.1 RC... this fixes a bug in the k8s integration tests
causing them to fail consistently.
On Mon, Mar 18, 2019 at 12:54 AM DB Tsai <db...@dbtsai.com.invalid> wrote:
> Rc6 was failed due to the critical bug Xiao mentioned. Rc8 was cut to
> include those fixes, but I ran into some nexus infra issues when I tried to
> upload the jars into Apache repo.
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/INFRA-18018
>
> The infra team reordered the key servers to prevent the timeout, and
> rebooted the server. I am running the build again to see if I can
> successfully upload Spark jars this time. Thanks!
>
> On Sun, Mar 17, 2019 at 2:32 PM Koert Kuipers <ko...@tresata.com> wrote:
>
>> hi all,
>> i am having a bit trouble following the RCs this time. is RC6 still the
>> one being voted on? i see on github v2.4.1-rc7 and v2.4.1-rc8 but no email
>> threads.
>> thanks!
>> best, koert
>>
>> On Mon, Mar 11, 2019 at 1:32 AM Xiao Li <li...@databricks.com> wrote:
>>
>>> ++ Ryan Blue. What is your opinion regarding this Parquet issue?
>>>
>>> Thanks,
>>>
>>> Xiao
>>>
>>> On Sun, Mar 10, 2019 at 9:58 PM Wenchen Fan <cl...@gmail.com> wrote:
>>>
>>>> Which version of Parquet has this bug? Maybe we can downgrade it.
>>>>
>>>> On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <ma...@clearstorydata.com>
>>>> wrote:
>>>>
>>>>> It worked in 2.3. We broke it with 2.4.0 and were informed of that
>>>>> regression late in the 2.4.0 release process. Since we didn't fix it before
>>>>> the 2.4.0 release, it should have been noted as a known issue. To now claim
>>>>> that there is no regression from 2.4.0 is a circular argument denying the
>>>>> existence of a known regression from 2.3.
>>>>>
>>>>> On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>>>
>>>>>> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading
>>>>>> that:
>>>>>>
>>>>>> - this is a Parquet-Avro version conflict thing
>>>>>> - a downstream app wants different versions of Parquet and Avro than
>>>>>> Spark uses, which triggers it
>>>>>> - it doesn't work in 2.4.0
>>>>>>
>>>>>> It's not a regression from 2.4.0, which is the immediate question.
>>>>>> There isn't even a Parquet fix available.
>>>>>> But I'm not even seeing why this is excuse-making?
>>>>>>
>>>>>> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Now wait... we created a regression in 2.4.0. Arguably, we should
>>>>>> have blocked that release until we had a fix; but the issue came up late in
>>>>>> the release process and it looks to me like there wasn't an adequate fix
>>>>>> immediately available, so we did something bad and released 2.4.0 with a
>>>>>> known regression. Saying that there is now no regression from 2.4 is
>>>>>> tautological and no excuse for not taking in a fix -- and it looks like
>>>>>> that fix has been waiting for months.
>>>>>>
>>>>>
>>>
>>> --
>>> [image:
>>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>>
>> --
> - DB Sent from my iPhone
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <db...@dbtsai.com.INVALID>.
Rc6 was failed due to the critical bug Xiao mentioned. Rc8 was cut to
include those fixes, but I ran into some nexus infra issues when I tried to
upload the jars into Apache repo.
https://issues.apache.org/jira/plugins/servlet/mobile#issue/INFRA-18018
The infra team reordered the key servers to prevent the timeout, and
rebooted the server. I am running the build again to see if I can
successfully upload Spark jars this time. Thanks!
On Sun, Mar 17, 2019 at 2:32 PM Koert Kuipers <ko...@tresata.com> wrote:
> hi all,
> i am having a bit trouble following the RCs this time. is RC6 still the
> one being voted on? i see on github v2.4.1-rc7 and v2.4.1-rc8 but no email
> threads.
> thanks!
> best, koert
>
> On Mon, Mar 11, 2019 at 1:32 AM Xiao Li <li...@databricks.com> wrote:
>
>> ++ Ryan Blue. What is your opinion regarding this Parquet issue?
>>
>> Thanks,
>>
>> Xiao
>>
>> On Sun, Mar 10, 2019 at 9:58 PM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> Which version of Parquet has this bug? Maybe we can downgrade it.
>>>
>>> On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <ma...@clearstorydata.com>
>>> wrote:
>>>
>>>> It worked in 2.3. We broke it with 2.4.0 and were informed of that
>>>> regression late in the 2.4.0 release process. Since we didn't fix it before
>>>> the 2.4.0 release, it should have been noted as a known issue. To now claim
>>>> that there is no regression from 2.4.0 is a circular argument denying the
>>>> existence of a known regression from 2.3.
>>>>
>>>> On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading
>>>>> that:
>>>>>
>>>>> - this is a Parquet-Avro version conflict thing
>>>>> - a downstream app wants different versions of Parquet and Avro than
>>>>> Spark uses, which triggers it
>>>>> - it doesn't work in 2.4.0
>>>>>
>>>>> It's not a regression from 2.4.0, which is the immediate question.
>>>>> There isn't even a Parquet fix available.
>>>>> But I'm not even seeing why this is excuse-making?
>>>>>
>>>>> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com>
>>>>> wrote:
>>>>> >
>>>>> > Now wait... we created a regression in 2.4.0. Arguably, we should
>>>>> have blocked that release until we had a fix; but the issue came up late in
>>>>> the release process and it looks to me like there wasn't an adequate fix
>>>>> immediately available, so we did something bad and released 2.4.0 with a
>>>>> known regression. Saying that there is now no regression from 2.4 is
>>>>> tautological and no excuse for not taking in a fix -- and it looks like
>>>>> that fix has been waiting for months.
>>>>>
>>>>
>>
>> --
>> [image:
>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>
> --
- DB Sent from my iPhone
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Koert Kuipers <ko...@tresata.com>.
hi all,
i am having a bit trouble following the RCs this time. is RC6 still the one
being voted on? i see on github v2.4.1-rc7 and v2.4.1-rc8 but no email
threads.
thanks!
best, koert
On Mon, Mar 11, 2019 at 1:32 AM Xiao Li <li...@databricks.com> wrote:
> ++ Ryan Blue. What is your opinion regarding this Parquet issue?
>
> Thanks,
>
> Xiao
>
> On Sun, Mar 10, 2019 at 9:58 PM Wenchen Fan <cl...@gmail.com> wrote:
>
>> Which version of Parquet has this bug? Maybe we can downgrade it.
>>
>> On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <ma...@clearstorydata.com>
>> wrote:
>>
>>> It worked in 2.3. We broke it with 2.4.0 and were informed of that
>>> regression late in the 2.4.0 release process. Since we didn't fix it before
>>> the 2.4.0 release, it should have been noted as a known issue. To now claim
>>> that there is no regression from 2.4.0 is a circular argument denying the
>>> existence of a known regression from 2.3.
>>>
>>> On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading
>>>> that:
>>>>
>>>> - this is a Parquet-Avro version conflict thing
>>>> - a downstream app wants different versions of Parquet and Avro than
>>>> Spark uses, which triggers it
>>>> - it doesn't work in 2.4.0
>>>>
>>>> It's not a regression from 2.4.0, which is the immediate question.
>>>> There isn't even a Parquet fix available.
>>>> But I'm not even seeing why this is excuse-making?
>>>>
>>>> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com>
>>>> wrote:
>>>> >
>>>> > Now wait... we created a regression in 2.4.0. Arguably, we should
>>>> have blocked that release until we had a fix; but the issue came up late in
>>>> the release process and it looks to me like there wasn't an adequate fix
>>>> immediately available, so we did something bad and released 2.4.0 with a
>>>> known regression. Saying that there is now no regression from 2.4 is
>>>> tautological and no excuse for not taking in a fix -- and it looks like
>>>> that fix has been waiting for months.
>>>>
>>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Xiao Li <li...@databricks.com>.
++ Ryan Blue. What is your opinion regarding this Parquet issue?
Thanks,
Xiao
On Sun, Mar 10, 2019 at 9:58 PM Wenchen Fan <cl...@gmail.com> wrote:
> Which version of Parquet has this bug? Maybe we can downgrade it.
>
> On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>
>> It worked in 2.3. We broke it with 2.4.0 and were informed of that
>> regression late in the 2.4.0 release process. Since we didn't fix it before
>> the 2.4.0 release, it should have been noted as a known issue. To now claim
>> that there is no regression from 2.4.0 is a circular argument denying the
>> existence of a known regression from 2.3.
>>
>> On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading
>>> that:
>>>
>>> - this is a Parquet-Avro version conflict thing
>>> - a downstream app wants different versions of Parquet and Avro than
>>> Spark uses, which triggers it
>>> - it doesn't work in 2.4.0
>>>
>>> It's not a regression from 2.4.0, which is the immediate question.
>>> There isn't even a Parquet fix available.
>>> But I'm not even seeing why this is excuse-making?
>>>
>>> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com>
>>> wrote:
>>> >
>>> > Now wait... we created a regression in 2.4.0. Arguably, we should have
>>> blocked that release until we had a fix; but the issue came up late in the
>>> release process and it looks to me like there wasn't an adequate fix
>>> immediately available, so we did something bad and released 2.4.0 with a
>>> known regression. Saying that there is now no regression from 2.4 is
>>> tautological and no excuse for not taking in a fix -- and it looks like
>>> that fix has been waiting for months.
>>>
>>
--
[image:
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Wenchen Fan <cl...@gmail.com>.
Which version of Parquet has this bug? Maybe we can downgrade it.
On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <ma...@clearstorydata.com>
wrote:
> It worked in 2.3. We broke it with 2.4.0 and were informed of that
> regression late in the 2.4.0 release process. Since we didn't fix it before
> the 2.4.0 release, it should have been noted as a known issue. To now claim
> that there is no regression from 2.4.0 is a circular argument denying the
> existence of a known regression from 2.3.
>
> On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sr...@gmail.com> wrote:
>
>> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:
>>
>> - this is a Parquet-Avro version conflict thing
>> - a downstream app wants different versions of Parquet and Avro than
>> Spark uses, which triggers it
>> - it doesn't work in 2.4.0
>>
>> It's not a regression from 2.4.0, which is the immediate question.
>> There isn't even a Parquet fix available.
>> But I'm not even seeing why this is excuse-making?
>>
>> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com>
>> wrote:
>> >
>> > Now wait... we created a regression in 2.4.0. Arguably, we should have
>> blocked that release until we had a fix; but the issue came up late in the
>> release process and it looks to me like there wasn't an adequate fix
>> immediately available, so we did something bad and released 2.4.0 with a
>> known regression. Saying that there is now no regression from 2.4 is
>> tautological and no excuse for not taking in a fix -- and it looks like
>> that fix has been waiting for months.
>>
>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Mark Hamstra <ma...@clearstorydata.com>.
It worked in 2.3. We broke it with 2.4.0 and were informed of that
regression late in the 2.4.0 release process. Since we didn't fix it before
the 2.4.0 release, it should have been noted as a known issue. To now claim
that there is no regression from 2.4.0 is a circular argument denying the
existence of a known regression from 2.3.
On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sr...@gmail.com> wrote:
> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:
>
> - this is a Parquet-Avro version conflict thing
> - a downstream app wants different versions of Parquet and Avro than
> Spark uses, which triggers it
> - it doesn't work in 2.4.0
>
> It's not a regression from 2.4.0, which is the immediate question.
> There isn't even a Parquet fix available.
> But I'm not even seeing why this is excuse-making?
>
> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com>
> wrote:
> >
> > Now wait... we created a regression in 2.4.0. Arguably, we should have
> blocked that release until we had a fix; but the issue came up late in the
> release process and it looks to me like there wasn't an adequate fix
> immediately available, so we did something bad and released 2.4.0 with a
> known regression. Saying that there is now no regression from 2.4 is
> tautological and no excuse for not taking in a fix -- and it looks like
> that fix has been waiting for months.
>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Sean Owen <sr...@gmail.com>.
From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:
- this is a Parquet-Avro version conflict thing
- a downstream app wants different versions of Parquet and Avro than
Spark uses, which triggers it
- it doesn't work in 2.4.0
It's not a regression from 2.4.0, which is the immediate question.
There isn't even a Parquet fix available.
But I'm not even seeing why this is excuse-making?
On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <ma...@clearstorydata.com> wrote:
>
> Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Now wait... we created a regression in 2.4.0. Arguably, we should have
blocked that release until we had a fix; but the issue came up late in the
release process and it looks to me like there wasn't an adequate fix
immediately available, so we did something bad and released 2.4.0 with a
known regression. Saying that there is now no regression from 2.4 is
tautological and no excuse for not taking in a fix -- and it looks like
that fix has been waiting for months.
On Sun, Mar 10, 2019 at 3:42 PM DB Tsai <db...@dbtsai.com.invalid> wrote:
> As we have many important fixes in 2.4 branch which we want to release
> asap, and this is is not a regression from Spark 2.4; as a result, 2.4.1
> will be not blocked by this.
>
> Sincerely,
>
> DB Tsai
> ----------------------------------------------------------
> Web: https://www.dbtsai.com
> PGP Key ID: 0359BC9965359766
>
>
> On Sun, Mar 10, 2019 at 3:08 PM Michael Heuer <he...@gmail.com> wrote:
>
>> Any chance we could get some movement on this for 2.4.1?
>>
>> https://issues.apache.org/jira/browse/SPARK-25588
>> https://github.com/apache/parquet-mr/pull/560
>>
>> It would require a new Parquet release, which would then need to be
>> picked up by Spark. We're dead in the water on 2.4.0 without a large
>> refactoring (remove all the RDD code paths for reading Avro stored in
>> Parquet).
>>
>> michael
>>
>>
>> On Mar 8, 2019, at 6:22 PM, Sean Owen <sr...@gmail.com> wrote:
>>
>> FWIW RC6 looked fine to me. Passed all tests, etc.
>>
>> On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <db...@dbtsai.com> wrote:
>>
>>> Sounds fair to me. I'll cut another rc7 when the PR is merged.
>>> Hopefully, this is the final rc. Thanks.
>>>
>>> Sincerely,
>>>
>>> DB Tsai
>>> ----------------------------------------------------------
>>> Web: https://www.dbtsai.com
>>> PGP Key ID: 42E5B25A8F7A82C1
>>>
>>>
>>> On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <li...@databricks.com> wrote:
>>>
>>>> It is common to hit this issue when driver and executors are different
>>>> object layout, but Spark might not return a wrong answer. It is very hard
>>>> to find out the root cause. Thus, I would suggest to include it in Spark
>>>> 2.4.1.
>>>>
>>>> On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <db...@dbtsai.com> wrote:
>>>>
>>>>> BTW, practically, is it common for users running into this bug when
>>>>> the driver and executors have different object layout?
>>>>>
>>>>> Sincerely,
>>>>>
>>>>> DB Tsai
>>>>> ----------------------------------------------------------
>>>>> Web: https://www.dbtsai.com
>>>>> PGP Key ID: 42E5B25A8F7A82C1
>>>>>
>>>>>
>>>>> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <db...@dbtsai.com> wrote:
>>>>>
>>>>>> Hi Xiao,
>>>>>>
>>>>>> I already cut rc7 and start the build process. If we definitely need
>>>>>> this fix, I can cut rc8. Let me know what you think.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <li...@databricks.com> wrote:
>>>>>>
>>>>>>> Hi, DB,
>>>>>>>
>>>>>>> Since this RC will fail, could you hold it until we fix
>>>>>>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I
>>>>>>> will submit a PR today. The PR is small and the risk is low. This is a
>>>>>>> correctness bug. It would be good to have it.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Xiao
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Since I can not find the commit of `Preparing development version
>>>>>>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
>>>>>>>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
>>>>>>>> this vote fails.
>>>>>>>>
>>>>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>>>>> Apple, Inc
>>>>>>>>
>>>>>>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's
>>>>>>>> very weird. It must be overwritten by a force push.
>>>>>>>> >
>>>>>>>> > DB Tsai | Siri Open Source Technologies [not a contribution]
>>>>>>>> | Apple, Inc
>>>>>>>> >
>>>>>>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> I was using `./do-release-docker.sh` to create a release. But
>>>>>>>> since the gpg validation fails couple times when the script tried to
>>>>>>>> publish the jars into Nexus, I re-ran the scripts multiple times without
>>>>>>>> creating a new rc. I was wondering if the script will overwrite the
>>>>>>>> v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>>>>>>> >>
>>>>>>>> >> Should we create a new rc7?
>>>>>>>> >>
>>>>>>>> >> DB Tsai | Siri Open Source Technologies [not a contribution]
>>>>>>>> | Apple, Inc
>>>>>>>> >>
>>>>>>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <
>>>>>>>> vanzin@cloudera.com.INVALID> wrote:
>>>>>>>> >>>
>>>>>>>> >>> I personally find it a little weird to not have the commit in
>>>>>>>> branch-2.4.
>>>>>>>> >>>
>>>>>>>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is
>>>>>>>> overwritten
>>>>>>>> >>> (e.g. accidentally) then you lose the reference to that commit,
>>>>>>>> and
>>>>>>>> >>> then the exact commit from which the rc was generated is lost.
>>>>>>>> >>>
>>>>>>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> That's weird. I see the commit but can't find it in the
>>>>>>>> branch. Was it pushed, or lost in a force push of 2.4 along the way? The
>>>>>>>> change is there, just under a different commit in the 2.4 branch.
>>>>>>>> >>>>
>>>>>>>> >>>> It doesn't necessarily invalidate the RC as it is a valid
>>>>>>>> public tagged commit and all that. I just want to be sure we do have the
>>>>>>>> code from that commit in these tatballs. It looks like it.
>>>>>>>> >>>>
>>>>>>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> Hi,
>>>>>>>> >>>>>
>>>>>>>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
>>>>>>>> branch-2.4. Release related commits I have seen so far were also part of
>>>>>>>> the branch.
>>>>>>>> >>>>>
>>>>>>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and
>>>>>>>> "Preparing development version 2.4.2-SNAPSHOT" commits were simply not
>>>>>>>> pushed to spark-2.4 just the tag itself was pushed. I dont know what is the
>>>>>>>> practice in such cases but one solution is to rebase branch-2.4 changes
>>>>>>>> after 3336a21 onto these commits and do a (sorry) force push. In this case
>>>>>>>> there is no impact on this RC.
>>>>>>>> >>>>>
>>>>>>>> >>>>> Best Regards,
>>>>>>>> >>>>>
>>>>>>>> >>>>> Misi
>>>>>>>> >>>>>
>>>>>>>> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019.
>>>>>>>> márc. 8., P, 1:15):
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Please vote on releasing the following candidate as Apache
>>>>>>>> Spark version 2.4.1.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> The vote is open until March 11 PST and passes if a majority
>>>>>>>> +1 PMC votes are cast, with
>>>>>>>> >>>>>> a minimum of 3 +1 votes.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>>>> >>>>>> [ ] -1 Do not release this package because ...
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> To learn more about Apache Spark, please see
>>>>>>>> http://spark.apache.org/
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
>>>>>>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> The release files, including signatures, digests, etc. can
>>>>>>>> be found at:
>>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> The staging repository for this release can be found at:
>>>>>>>> >>>>>>
>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> The documentation corresponding to this release can be found
>>>>>>>> at:
>>>>>>>> >>>>>>
>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
>>>>>>>> following URL:
>>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> FAQ
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> =========================
>>>>>>>> >>>>>> How can I help test this release?
>>>>>>>> >>>>>> =========================
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> If you are a Spark user, you can help us test this release
>>>>>>>> by taking
>>>>>>>> >>>>>> an existing Spark workload and running on this release
>>>>>>>> candidate, then
>>>>>>>> >>>>>> reporting any regressions.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> If you're working in PySpark you can set up a virtual env
>>>>>>>> and install
>>>>>>>> >>>>>> the current RC and see if anything important breaks, in the
>>>>>>>> Java/Scala
>>>>>>>> >>>>>> you can add the staging repository to your projects
>>>>>>>> resolvers and test
>>>>>>>> >>>>>> with the RC (make sure to clean up the artifact cache
>>>>>>>> before/after so
>>>>>>>> >>>>>> you don't end up building with a out of date RC going
>>>>>>>> forward).
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> ===========================================
>>>>>>>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>>>> >>>>>> ===========================================
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> The current list of open tickets targeted at 2.4.1 can be
>>>>>>>> found at:
>>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search
>>>>>>>> for "Target Version/s" = 2.4.1
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Committers should look at those and triage. Extremely
>>>>>>>> important bug
>>>>>>>> >>>>>> fixes, documentation, and API tweaks that impact
>>>>>>>> compatibility should
>>>>>>>> >>>>>> be worked on immediately. Everything else please retarget to
>>>>>>>> an
>>>>>>>> >>>>>> appropriate release.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> ==================
>>>>>>>> >>>>>> But my bug isn't fixed?
>>>>>>>> >>>>>> ==================
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> In order to make timely releases, we will typically not hold
>>>>>>>> the
>>>>>>>> >>>>>> release unless the bug in question is a regression from the
>>>>>>>> previous
>>>>>>>> >>>>>> release. That being said, if there is something which is a
>>>>>>>> regression
>>>>>>>> >>>>>> that has not been correctly targeted please ping me or a
>>>>>>>> committer to
>>>>>>>> >>>>>> help target the issue.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a
>>>>>>>> contribution] | Apple, Inc
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> >>>>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> --
>>>>>>>> >>> Marcelo
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> >>>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> >>
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> <SA...@2X.jpg>
>>>>>>>
>>>>>> --
>>>>>> - DB Sent from my iPhone
>>>>>>
>>>>>
>>>>
>>>> --
>>>> <SA...@2X.jpg>
>>>>
>>>
>>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <db...@dbtsai.com.INVALID>.
As we have many important fixes in 2.4 branch which we want to release
asap, and this is is not a regression from Spark 2.4; as a result, 2.4.1
will be not blocked by this.
Sincerely,
DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0359BC9965359766
On Sun, Mar 10, 2019 at 3:08 PM Michael Heuer <he...@gmail.com> wrote:
> Any chance we could get some movement on this for 2.4.1?
>
> https://issues.apache.org/jira/browse/SPARK-25588
> https://github.com/apache/parquet-mr/pull/560
>
> It would require a new Parquet release, which would then need to be picked
> up by Spark. We're dead in the water on 2.4.0 without a large refactoring
> (remove all the RDD code paths for reading Avro stored in Parquet).
>
> michael
>
>
> On Mar 8, 2019, at 6:22 PM, Sean Owen <sr...@gmail.com> wrote:
>
> FWIW RC6 looked fine to me. Passed all tests, etc.
>
> On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <db...@dbtsai.com> wrote:
>
>> Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully,
>> this is the final rc. Thanks.
>>
>> Sincerely,
>>
>> DB Tsai
>> ----------------------------------------------------------
>> Web: https://www.dbtsai.com
>> PGP Key ID: 42E5B25A8F7A82C1
>>
>>
>> On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <li...@databricks.com> wrote:
>>
>>> It is common to hit this issue when driver and executors are different
>>> object layout, but Spark might not return a wrong answer. It is very hard
>>> to find out the root cause. Thus, I would suggest to include it in Spark
>>> 2.4.1.
>>>
>>> On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <db...@dbtsai.com> wrote:
>>>
>>>> BTW, practically, is it common for users running into this bug when the
>>>> driver and executors have different object layout?
>>>>
>>>> Sincerely,
>>>>
>>>> DB Tsai
>>>> ----------------------------------------------------------
>>>> Web: https://www.dbtsai.com
>>>> PGP Key ID: 42E5B25A8F7A82C1
>>>>
>>>>
>>>> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <db...@dbtsai.com> wrote:
>>>>
>>>>> Hi Xiao,
>>>>>
>>>>> I already cut rc7 and start the build process. If we definitely need
>>>>> this fix, I can cut rc8. Let me know what you think.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <li...@databricks.com> wrote:
>>>>>
>>>>>> Hi, DB,
>>>>>>
>>>>>> Since this RC will fail, could you hold it until we fix
>>>>>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I
>>>>>> will submit a PR today. The PR is small and the risk is low. This is a
>>>>>> correctness bug. It would be good to have it.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Xiao
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> Since I can not find the commit of `Preparing development version
>>>>>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
>>>>>>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
>>>>>>> this vote fails.
>>>>>>>
>>>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>>>> Apple, Inc
>>>>>>>
>>>>>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's
>>>>>>> very weird. It must be overwritten by a force push.
>>>>>>> >
>>>>>>> > DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>>>> Apple, Inc
>>>>>>> >
>>>>>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> I was using `./do-release-docker.sh` to create a release. But
>>>>>>> since the gpg validation fails couple times when the script tried to
>>>>>>> publish the jars into Nexus, I re-ran the scripts multiple times without
>>>>>>> creating a new rc. I was wondering if the script will overwrite the
>>>>>>> v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>>>>>> >>
>>>>>>> >> Should we create a new rc7?
>>>>>>> >>
>>>>>>> >> DB Tsai | Siri Open Source Technologies [not a contribution]
>>>>>>> | Apple, Inc
>>>>>>> >>
>>>>>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <
>>>>>>> vanzin@cloudera.com.INVALID> wrote:
>>>>>>> >>>
>>>>>>> >>> I personally find it a little weird to not have the commit in
>>>>>>> branch-2.4.
>>>>>>> >>>
>>>>>>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is
>>>>>>> overwritten
>>>>>>> >>> (e.g. accidentally) then you lose the reference to that commit,
>>>>>>> and
>>>>>>> >>> then the exact commit from which the rc was generated is lost.
>>>>>>> >>>
>>>>>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> That's weird. I see the commit but can't find it in the branch.
>>>>>>> Was it pushed, or lost in a force push of 2.4 along the way? The change is
>>>>>>> there, just under a different commit in the 2.4 branch.
>>>>>>> >>>>
>>>>>>> >>>> It doesn't necessarily invalidate the RC as it is a valid
>>>>>>> public tagged commit and all that. I just want to be sure we do have the
>>>>>>> code from that commit in these tatballs. It looks like it.
>>>>>>> >>>>
>>>>>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Hi,
>>>>>>> >>>>>
>>>>>>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
>>>>>>> branch-2.4. Release related commits I have seen so far were also part of
>>>>>>> the branch.
>>>>>>> >>>>>
>>>>>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and
>>>>>>> "Preparing development version 2.4.2-SNAPSHOT" commits were simply not
>>>>>>> pushed to spark-2.4 just the tag itself was pushed. I dont know what is the
>>>>>>> practice in such cases but one solution is to rebase branch-2.4 changes
>>>>>>> after 3336a21 onto these commits and do a (sorry) force push. In this case
>>>>>>> there is no impact on this RC.
>>>>>>> >>>>>
>>>>>>> >>>>> Best Regards,
>>>>>>> >>>>>
>>>>>>> >>>>> Misi
>>>>>>> >>>>>
>>>>>>> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019.
>>>>>>> márc. 8., P, 1:15):
>>>>>>> >>>>>>
>>>>>>> >>>>>> Please vote on releasing the following candidate as Apache
>>>>>>> Spark version 2.4.1.
>>>>>>> >>>>>>
>>>>>>> >>>>>> The vote is open until March 11 PST and passes if a majority
>>>>>>> +1 PMC votes are cast, with
>>>>>>> >>>>>> a minimum of 3 +1 votes.
>>>>>>> >>>>>>
>>>>>>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>>> >>>>>> [ ] -1 Do not release this package because ...
>>>>>>> >>>>>>
>>>>>>> >>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>> >>>>>>
>>>>>>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
>>>>>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>> >>>>>>
>>>>>>> >>>>>> The release files, including signatures, digests, etc. can be
>>>>>>> found at:
>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>> >>>>>>
>>>>>>> >>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>> >>>>>>
>>>>>>> >>>>>> The staging repository for this release can be found at:
>>>>>>> >>>>>>
>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>> >>>>>>
>>>>>>> >>>>>> The documentation corresponding to this release can be found
>>>>>>> at:
>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>> >>>>>>
>>>>>>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
>>>>>>> following URL:
>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>> >>>>>>
>>>>>>> >>>>>> FAQ
>>>>>>> >>>>>>
>>>>>>> >>>>>> =========================
>>>>>>> >>>>>> How can I help test this release?
>>>>>>> >>>>>> =========================
>>>>>>> >>>>>>
>>>>>>> >>>>>> If you are a Spark user, you can help us test this release by
>>>>>>> taking
>>>>>>> >>>>>> an existing Spark workload and running on this release
>>>>>>> candidate, then
>>>>>>> >>>>>> reporting any regressions.
>>>>>>> >>>>>>
>>>>>>> >>>>>> If you're working in PySpark you can set up a virtual env and
>>>>>>> install
>>>>>>> >>>>>> the current RC and see if anything important breaks, in the
>>>>>>> Java/Scala
>>>>>>> >>>>>> you can add the staging repository to your projects resolvers
>>>>>>> and test
>>>>>>> >>>>>> with the RC (make sure to clean up the artifact cache
>>>>>>> before/after so
>>>>>>> >>>>>> you don't end up building with a out of date RC going
>>>>>>> forward).
>>>>>>> >>>>>>
>>>>>>> >>>>>> ===========================================
>>>>>>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>>> >>>>>> ===========================================
>>>>>>> >>>>>>
>>>>>>> >>>>>> The current list of open tickets targeted at 2.4.1 can be
>>>>>>> found at:
>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search for
>>>>>>> "Target Version/s" = 2.4.1
>>>>>>> >>>>>>
>>>>>>> >>>>>> Committers should look at those and triage. Extremely
>>>>>>> important bug
>>>>>>> >>>>>> fixes, documentation, and API tweaks that impact
>>>>>>> compatibility should
>>>>>>> >>>>>> be worked on immediately. Everything else please retarget to
>>>>>>> an
>>>>>>> >>>>>> appropriate release.
>>>>>>> >>>>>>
>>>>>>> >>>>>> ==================
>>>>>>> >>>>>> But my bug isn't fixed?
>>>>>>> >>>>>> ==================
>>>>>>> >>>>>>
>>>>>>> >>>>>> In order to make timely releases, we will typically not hold
>>>>>>> the
>>>>>>> >>>>>> release unless the bug in question is a regression from the
>>>>>>> previous
>>>>>>> >>>>>> release. That being said, if there is something which is a
>>>>>>> regression
>>>>>>> >>>>>> that has not been correctly targeted please ping me or a
>>>>>>> committer to
>>>>>>> >>>>>> help target the issue.
>>>>>>> >>>>>>
>>>>>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a
>>>>>>> contribution] | Apple, Inc
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >>>>>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> --
>>>>>>> >>> Marcelo
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >>
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> <SA...@2X.jpg>
>>>>>>
>>>>> --
>>>>> - DB Sent from my iPhone
>>>>>
>>>>
>>>
>>> --
>>> <SA...@2X.jpg>
>>>
>>
>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Michael Heuer <he...@gmail.com>.
Any chance we could get some movement on this for 2.4.1?
https://issues.apache.org/jira/browse/SPARK-25588 <https://issues.apache.org/jira/browse/SPARK-25588>
https://github.com/apache/parquet-mr/pull/560 <https://github.com/apache/parquet-mr/pull/560>
It would require a new Parquet release, which would then need to be picked up by Spark. We're dead in the water on 2.4.0 without a large refactoring (remove all the RDD code paths for reading Avro stored in Parquet).
michael
> On Mar 8, 2019, at 6:22 PM, Sean Owen <sr...@gmail.com> wrote:
>
> FWIW RC6 looked fine to me. Passed all tests, etc.
>
> On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <dbtsai@dbtsai.com <ma...@dbtsai.com>> wrote:
> Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully, this is the final rc. Thanks.
>
> Sincerely,
>
> DB Tsai
> ----------------------------------------------------------
> Web: https://www.dbtsai.com <https://www.dbtsai.com/>
> PGP Key ID: 42E5B25A8F7A82C1
>
>
> On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <lixiao@databricks.com <ma...@databricks.com>> wrote:
> It is common to hit this issue when driver and executors are different object layout, but Spark might not return a wrong answer. It is very hard to find out the root cause. Thus, I would suggest to include it in Spark 2.4.1.
>
> On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <dbtsai@dbtsai.com <ma...@dbtsai.com>> wrote:
> BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?
>
> Sincerely,
>
> DB Tsai
> ----------------------------------------------------------
> Web: https://www.dbtsai.com <https://www.dbtsai.com/>
> PGP Key ID: 42E5B25A8F7A82C1
>
>
> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <dbtsai@dbtsai.com <ma...@dbtsai.com>> wrote:
> Hi Xiao,
>
> I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.
>
> Thanks,
>
> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <lixiao@databricks.com <ma...@databricks.com>> wrote:
> Hi, DB,
>
> Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097 <https://issues.apache.org/jira/browse/SPARK-27097>? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it.
>
> Thanks,
>
> Xiao
>
>
>
>
> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid> wrote:
> Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.
>
> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>
> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
> >
> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
> >
> > DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> >
> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
> >>
> >> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
> >>
> >> Should we create a new rc7?
> >>
> >> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> >>
> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <va...@cloudera.com.INVALID> wrote:
> >>>
> >>> I personally find it a little weird to not have the commit in branch-2.4.
> >>>
> >>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
> >>> (e.g. accidentally) then you lose the reference to that commit, and
> >>> then the exact commit from which the rc was generated is lost.
> >>>
> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <srowen@gmail.com <ma...@gmail.com>> wrote:
> >>>>
> >>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
> >>>>
> >>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
> >>>>
> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <misutoth@gmail.com <ma...@gmail.com>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
> >>>>>
> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
> >>>>>
> >>>>> Best Regards,
> >>>>>
> >>>>> Misi
> >>>>>
> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P, 1:15):
> >>>>>>
> >>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
> >>>>>>
> >>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
> >>>>>> a minimum of 3 +1 votes.
> >>>>>>
> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
> >>>>>> [ ] -1 Do not release this package because ...
> >>>>>>
> >>>>>> To learn more about Apache Spark, please see http://spark.apache.org/ <http://spark.apache.org/>
> >>>>>>
> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6 <https://github.com/apache/spark/tree/v2.4.1-rc6>
> >>>>>>
> >>>>>> The release files, including signatures, digests, etc. can be found at:
> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/ <https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/>
> >>>>>>
> >>>>>> Signatures used for Spark RCs can be found in this file:
> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS <https://dist.apache.org/repos/dist/dev/spark/KEYS>
> >>>>>>
> >>>>>> The staging repository for this release can be found at:
> >>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/ <https://repository.apache.org/content/repositories/orgapachespark-1308/>
> >>>>>>
> >>>>>> The documentation corresponding to this release can be found at:
> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/ <https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/>
> >>>>>>
> >>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1 <https://issues.apache.org/jira/projects/SPARK/versions/2.4.1>
> >>>>>>
> >>>>>> FAQ
> >>>>>>
> >>>>>> =========================
> >>>>>> How can I help test this release?
> >>>>>> =========================
> >>>>>>
> >>>>>> If you are a Spark user, you can help us test this release by taking
> >>>>>> an existing Spark workload and running on this release candidate, then
> >>>>>> reporting any regressions.
> >>>>>>
> >>>>>> If you're working in PySpark you can set up a virtual env and install
> >>>>>> the current RC and see if anything important breaks, in the Java/Scala
> >>>>>> you can add the staging repository to your projects resolvers and test
> >>>>>> with the RC (make sure to clean up the artifact cache before/after so
> >>>>>> you don't end up building with a out of date RC going forward).
> >>>>>>
> >>>>>> ===========================================
> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
> >>>>>> ===========================================
> >>>>>>
> >>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
> >>>>>> https://issues.apache.org/jira/projects/SPARK <https://issues.apache.org/jira/projects/SPARK> and search for "Target Version/s" = 2.4.1
> >>>>>>
> >>>>>> Committers should look at those and triage. Extremely important bug
> >>>>>> fixes, documentation, and API tweaks that impact compatibility should
> >>>>>> be worked on immediately. Everything else please retarget to an
> >>>>>> appropriate release.
> >>>>>>
> >>>>>> ==================
> >>>>>> But my bug isn't fixed?
> >>>>>> ==================
> >>>>>>
> >>>>>> In order to make timely releases, we will typically not hold the
> >>>>>> release unless the bug in question is a regression from the previous
> >>>>>> release. That being said, if there is something which is a regression
> >>>>>> that has not been correctly targeted please ping me or a committer to
> >>>>>> help target the issue.
> >>>>>>
> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> >>>>>>
> >>>
> >>>
> >>> --
> >>> Marcelo
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>
>
>
> --
> <SA...@2X.jpg>
> --
> - DB Sent from my iPhone
>
>
> --
> <SA...@2X.jpg>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Sean Owen <sr...@gmail.com>.
FWIW RC6 looked fine to me. Passed all tests, etc.
On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <db...@dbtsai.com> wrote:
> Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully,
> this is the final rc. Thanks.
>
> Sincerely,
>
> DB Tsai
> ----------------------------------------------------------
> Web: https://www.dbtsai.com
> PGP Key ID: 42E5B25A8F7A82C1
>
>
> On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <li...@databricks.com> wrote:
>
>> It is common to hit this issue when driver and executors are different
>> object layout, but Spark might not return a wrong answer. It is very hard
>> to find out the root cause. Thus, I would suggest to include it in Spark
>> 2.4.1.
>>
>> On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <db...@dbtsai.com> wrote:
>>
>>> BTW, practically, is it common for users running into this bug when the
>>> driver and executors have different object layout?
>>>
>>> Sincerely,
>>>
>>> DB Tsai
>>> ----------------------------------------------------------
>>> Web: https://www.dbtsai.com
>>> PGP Key ID: 42E5B25A8F7A82C1
>>>
>>>
>>> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <db...@dbtsai.com> wrote:
>>>
>>>> Hi Xiao,
>>>>
>>>> I already cut rc7 and start the build process. If we definitely need
>>>> this fix, I can cut rc8. Let me know what you think.
>>>>
>>>> Thanks,
>>>>
>>>> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <li...@databricks.com> wrote:
>>>>
>>>>> Hi, DB,
>>>>>
>>>>> Since this RC will fail, could you hold it until we fix
>>>>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I
>>>>> will submit a PR today. The PR is small and the risk is low. This is a
>>>>> correctness bug. It would be good to have it.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Xiao
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> Since I can not find the commit of `Preparing development version
>>>>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
>>>>>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
>>>>>> this vote fails.
>>>>>>
>>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>>> Apple, Inc
>>>>>>
>>>>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>>> wrote:
>>>>>> >
>>>>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's
>>>>>> very weird. It must be overwritten by a force push.
>>>>>> >
>>>>>> > DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>>> Apple, Inc
>>>>>> >
>>>>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> I was using `./do-release-docker.sh` to create a release. But
>>>>>> since the gpg validation fails couple times when the script tried to
>>>>>> publish the jars into Nexus, I re-ran the scripts multiple times without
>>>>>> creating a new rc. I was wondering if the script will overwrite the
>>>>>> v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>>>>> >>
>>>>>> >> Should we create a new rc7?
>>>>>> >>
>>>>>> >> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>>> Apple, Inc
>>>>>> >>
>>>>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin
>>>>>> <va...@cloudera.com.INVALID> wrote:
>>>>>> >>>
>>>>>> >>> I personally find it a little weird to not have the commit in
>>>>>> branch-2.4.
>>>>>> >>>
>>>>>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is
>>>>>> overwritten
>>>>>> >>> (e.g. accidentally) then you lose the reference to that commit,
>>>>>> and
>>>>>> >>> then the exact commit from which the rc was generated is lost.
>>>>>> >>>
>>>>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> That's weird. I see the commit but can't find it in the branch.
>>>>>> Was it pushed, or lost in a force push of 2.4 along the way? The change is
>>>>>> there, just under a different commit in the 2.4 branch.
>>>>>> >>>>
>>>>>> >>>> It doesn't necessarily invalidate the RC as it is a valid public
>>>>>> tagged commit and all that. I just want to be sure we do have the code from
>>>>>> that commit in these tatballs. It looks like it.
>>>>>> >>>>
>>>>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com>
>>>>>> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Hi,
>>>>>> >>>>>
>>>>>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
>>>>>> branch-2.4. Release related commits I have seen so far were also part of
>>>>>> the branch.
>>>>>> >>>>>
>>>>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing
>>>>>> development version 2.4.2-SNAPSHOT" commits were simply not pushed to
>>>>>> spark-2.4 just the tag itself was pushed. I dont know what is the practice
>>>>>> in such cases but one solution is to rebase branch-2.4 changes after
>>>>>> 3336a21 onto these commits and do a (sorry) force push. In this case there
>>>>>> is no impact on this RC.
>>>>>> >>>>>
>>>>>> >>>>> Best Regards,
>>>>>> >>>>>
>>>>>> >>>>> Misi
>>>>>> >>>>>
>>>>>> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019.
>>>>>> márc. 8., P, 1:15):
>>>>>> >>>>>>
>>>>>> >>>>>> Please vote on releasing the following candidate as Apache
>>>>>> Spark version 2.4.1.
>>>>>> >>>>>>
>>>>>> >>>>>> The vote is open until March 11 PST and passes if a majority
>>>>>> +1 PMC votes are cast, with
>>>>>> >>>>>> a minimum of 3 +1 votes.
>>>>>> >>>>>>
>>>>>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> >>>>>> [ ] -1 Do not release this package because ...
>>>>>> >>>>>>
>>>>>> >>>>>> To learn more about Apache Spark, please see
>>>>>> http://spark.apache.org/
>>>>>> >>>>>>
>>>>>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
>>>>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>> >>>>>>
>>>>>> >>>>>> The release files, including signatures, digests, etc. can be
>>>>>> found at:
>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>> >>>>>>
>>>>>> >>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>> >>>>>>
>>>>>> >>>>>> The staging repository for this release can be found at:
>>>>>> >>>>>>
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>> >>>>>>
>>>>>> >>>>>> The documentation corresponding to this release can be found
>>>>>> at:
>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>> >>>>>>
>>>>>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
>>>>>> following URL:
>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>> >>>>>>
>>>>>> >>>>>> FAQ
>>>>>> >>>>>>
>>>>>> >>>>>> =========================
>>>>>> >>>>>> How can I help test this release?
>>>>>> >>>>>> =========================
>>>>>> >>>>>>
>>>>>> >>>>>> If you are a Spark user, you can help us test this release by
>>>>>> taking
>>>>>> >>>>>> an existing Spark workload and running on this release
>>>>>> candidate, then
>>>>>> >>>>>> reporting any regressions.
>>>>>> >>>>>>
>>>>>> >>>>>> If you're working in PySpark you can set up a virtual env and
>>>>>> install
>>>>>> >>>>>> the current RC and see if anything important breaks, in the
>>>>>> Java/Scala
>>>>>> >>>>>> you can add the staging repository to your projects resolvers
>>>>>> and test
>>>>>> >>>>>> with the RC (make sure to clean up the artifact cache
>>>>>> before/after so
>>>>>> >>>>>> you don't end up building with a out of date RC going forward).
>>>>>> >>>>>>
>>>>>> >>>>>> ===========================================
>>>>>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> >>>>>> ===========================================
>>>>>> >>>>>>
>>>>>> >>>>>> The current list of open tickets targeted at 2.4.1 can be
>>>>>> found at:
>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search for
>>>>>> "Target Version/s" = 2.4.1
>>>>>> >>>>>>
>>>>>> >>>>>> Committers should look at those and triage. Extremely
>>>>>> important bug
>>>>>> >>>>>> fixes, documentation, and API tweaks that impact compatibility
>>>>>> should
>>>>>> >>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> >>>>>> appropriate release.
>>>>>> >>>>>>
>>>>>> >>>>>> ==================
>>>>>> >>>>>> But my bug isn't fixed?
>>>>>> >>>>>> ==================
>>>>>> >>>>>>
>>>>>> >>>>>> In order to make timely releases, we will typically not hold
>>>>>> the
>>>>>> >>>>>> release unless the bug in question is a regression from the
>>>>>> previous
>>>>>> >>>>>> release. That being said, if there is something which is a
>>>>>> regression
>>>>>> >>>>>> that has not been correctly targeted please ping me or a
>>>>>> committer to
>>>>>> >>>>>> help target the issue.
>>>>>> >>>>>>
>>>>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a
>>>>>> contribution] | Apple, Inc
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>> >>>>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> Marcelo
>>>>>> >>>
>>>>>> >>>
>>>>>> ---------------------------------------------------------------------
>>>>>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> ---------------------------------------------------------------------
>>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> ---------------------------------------------------------------------
>>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>> >
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> [image:
>>>>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>>>>
>>>> --
>>>> - DB Sent from my iPhone
>>>>
>>>
>>
>> --
>> [image:
>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>
>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <db...@dbtsai.com.INVALID>.
Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully,
this is the final rc. Thanks.
Sincerely,
DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1
On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <li...@databricks.com> wrote:
> It is common to hit this issue when driver and executors are different
> object layout, but Spark might not return a wrong answer. It is very hard
> to find out the root cause. Thus, I would suggest to include it in Spark
> 2.4.1.
>
> On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <db...@dbtsai.com> wrote:
>
>> BTW, practically, is it common for users running into this bug when the
>> driver and executors have different object layout?
>>
>> Sincerely,
>>
>> DB Tsai
>> ----------------------------------------------------------
>> Web: https://www.dbtsai.com
>> PGP Key ID: 42E5B25A8F7A82C1
>>
>>
>> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <db...@dbtsai.com> wrote:
>>
>>> Hi Xiao,
>>>
>>> I already cut rc7 and start the build process. If we definitely need
>>> this fix, I can cut rc8. Let me know what you think.
>>>
>>> Thanks,
>>>
>>> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <li...@databricks.com> wrote:
>>>
>>>> Hi, DB,
>>>>
>>>> Since this RC will fail, could you hold it until we fix
>>>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I
>>>> will submit a PR today. The PR is small and the risk is low. This is a
>>>> correctness bug. It would be good to have it.
>>>>
>>>> Thanks,
>>>>
>>>> Xiao
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid>
>>>> wrote:
>>>>
>>>>> Since I can not find the commit of `Preparing development version
>>>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
>>>>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
>>>>> this vote fails.
>>>>>
>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>> Apple, Inc
>>>>>
>>>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>> wrote:
>>>>> >
>>>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very
>>>>> weird. It must be overwritten by a force push.
>>>>> >
>>>>> > DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>> Apple, Inc
>>>>> >
>>>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID>
>>>>> wrote:
>>>>> >>
>>>>> >> I was using `./do-release-docker.sh` to create a release. But since
>>>>> the gpg validation fails couple times when the script tried to publish the
>>>>> jars into Nexus, I re-ran the scripts multiple times without creating a new
>>>>> rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag
>>>>> instead of using the same commit causing this issue.
>>>>> >>
>>>>> >> Should we create a new rc7?
>>>>> >>
>>>>> >> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>>> Apple, Inc
>>>>> >>
>>>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin
>>>>> <va...@cloudera.com.INVALID> wrote:
>>>>> >>>
>>>>> >>> I personally find it a little weird to not have the commit in
>>>>> branch-2.4.
>>>>> >>>
>>>>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is
>>>>> overwritten
>>>>> >>> (e.g. accidentally) then you lose the reference to that commit, and
>>>>> >>> then the exact commit from which the rc was generated is lost.
>>>>> >>>
>>>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> That's weird. I see the commit but can't find it in the branch.
>>>>> Was it pushed, or lost in a force push of 2.4 along the way? The change is
>>>>> there, just under a different commit in the 2.4 branch.
>>>>> >>>>
>>>>> >>>> It doesn't necessarily invalidate the RC as it is a valid public
>>>>> tagged commit and all that. I just want to be sure we do have the code from
>>>>> that commit in these tatballs. It looks like it.
>>>>> >>>>
>>>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Hi,
>>>>> >>>>>
>>>>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
>>>>> branch-2.4. Release related commits I have seen so far were also part of
>>>>> the branch.
>>>>> >>>>>
>>>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing
>>>>> development version 2.4.2-SNAPSHOT" commits were simply not pushed to
>>>>> spark-2.4 just the tag itself was pushed. I dont know what is the practice
>>>>> in such cases but one solution is to rebase branch-2.4 changes after
>>>>> 3336a21 onto these commits and do a (sorry) force push. In this case there
>>>>> is no impact on this RC.
>>>>> >>>>>
>>>>> >>>>> Best Regards,
>>>>> >>>>>
>>>>> >>>>> Misi
>>>>> >>>>>
>>>>> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019.
>>>>> márc. 8., P, 1:15):
>>>>> >>>>>>
>>>>> >>>>>> Please vote on releasing the following candidate as Apache
>>>>> Spark version 2.4.1.
>>>>> >>>>>>
>>>>> >>>>>> The vote is open until March 11 PST and passes if a majority +1
>>>>> PMC votes are cast, with
>>>>> >>>>>> a minimum of 3 +1 votes.
>>>>> >>>>>>
>>>>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>> >>>>>> [ ] -1 Do not release this package because ...
>>>>> >>>>>>
>>>>> >>>>>> To learn more about Apache Spark, please see
>>>>> http://spark.apache.org/
>>>>> >>>>>>
>>>>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
>>>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>> >>>>>>
>>>>> >>>>>> The release files, including signatures, digests, etc. can be
>>>>> found at:
>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>> >>>>>>
>>>>> >>>>>> Signatures used for Spark RCs can be found in this file:
>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>> >>>>>>
>>>>> >>>>>> The staging repository for this release can be found at:
>>>>> >>>>>>
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>> >>>>>>
>>>>> >>>>>> The documentation corresponding to this release can be found at:
>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>> >>>>>>
>>>>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
>>>>> following URL:
>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>> >>>>>>
>>>>> >>>>>> FAQ
>>>>> >>>>>>
>>>>> >>>>>> =========================
>>>>> >>>>>> How can I help test this release?
>>>>> >>>>>> =========================
>>>>> >>>>>>
>>>>> >>>>>> If you are a Spark user, you can help us test this release by
>>>>> taking
>>>>> >>>>>> an existing Spark workload and running on this release
>>>>> candidate, then
>>>>> >>>>>> reporting any regressions.
>>>>> >>>>>>
>>>>> >>>>>> If you're working in PySpark you can set up a virtual env and
>>>>> install
>>>>> >>>>>> the current RC and see if anything important breaks, in the
>>>>> Java/Scala
>>>>> >>>>>> you can add the staging repository to your projects resolvers
>>>>> and test
>>>>> >>>>>> with the RC (make sure to clean up the artifact cache
>>>>> before/after so
>>>>> >>>>>> you don't end up building with a out of date RC going forward).
>>>>> >>>>>>
>>>>> >>>>>> ===========================================
>>>>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>> >>>>>> ===========================================
>>>>> >>>>>>
>>>>> >>>>>> The current list of open tickets targeted at 2.4.1 can be found
>>>>> at:
>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search for
>>>>> "Target Version/s" = 2.4.1
>>>>> >>>>>>
>>>>> >>>>>> Committers should look at those and triage. Extremely important
>>>>> bug
>>>>> >>>>>> fixes, documentation, and API tweaks that impact compatibility
>>>>> should
>>>>> >>>>>> be worked on immediately. Everything else please retarget to an
>>>>> >>>>>> appropriate release.
>>>>> >>>>>>
>>>>> >>>>>> ==================
>>>>> >>>>>> But my bug isn't fixed?
>>>>> >>>>>> ==================
>>>>> >>>>>>
>>>>> >>>>>> In order to make timely releases, we will typically not hold the
>>>>> >>>>>> release unless the bug in question is a regression from the
>>>>> previous
>>>>> >>>>>> release. That being said, if there is something which is a
>>>>> regression
>>>>> >>>>>> that has not been correctly targeted please ping me or a
>>>>> committer to
>>>>> >>>>>> help target the issue.
>>>>> >>>>>>
>>>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution]
>>>>> | Apple, Inc
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >>>>>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Marcelo
>>>>> >>>
>>>>> >>>
>>>>> ---------------------------------------------------------------------
>>>>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> ---------------------------------------------------------------------
>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >>
>>>>> >
>>>>> >
>>>>> > ---------------------------------------------------------------------
>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>
>>>> --
>>>> [image:
>>>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>>>
>>> --
>>> - DB Sent from my iPhone
>>>
>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Xiao Li <li...@databricks.com>.
It is common to hit this issue when driver and executors are different
object layout, but Spark might not return a wrong answer. It is very hard
to find out the root cause. Thus, I would suggest to include it in Spark
2.4.1.
On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <db...@dbtsai.com> wrote:
> BTW, practically, is it common for users running into this bug when the
> driver and executors have different object layout?
>
> Sincerely,
>
> DB Tsai
> ----------------------------------------------------------
> Web: https://www.dbtsai.com
> PGP Key ID: 42E5B25A8F7A82C1
>
>
> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <db...@dbtsai.com> wrote:
>
>> Hi Xiao,
>>
>> I already cut rc7 and start the build process. If we definitely need this
>> fix, I can cut rc8. Let me know what you think.
>>
>> Thanks,
>>
>> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <li...@databricks.com> wrote:
>>
>>> Hi, DB,
>>>
>>> Since this RC will fail, could you hold it until we fix
>>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I
>>> will submit a PR today. The PR is small and the risk is low. This is a
>>> correctness bug. It would be good to have it.
>>>
>>> Thanks,
>>>
>>> Xiao
>>>
>>>
>>>
>>>
>>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid>
>>> wrote:
>>>
>>>> Since I can not find the commit of `Preparing development version
>>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
>>>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
>>>> this vote fails.
>>>>
>>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>> Apple, Inc
>>>>
>>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID>
>>>> wrote:
>>>> >
>>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very
>>>> weird. It must be overwritten by a force push.
>>>> >
>>>> > DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>> Apple, Inc
>>>> >
>>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID>
>>>> wrote:
>>>> >>
>>>> >> I was using `./do-release-docker.sh` to create a release. But since
>>>> the gpg validation fails couple times when the script tried to publish the
>>>> jars into Nexus, I re-ran the scripts multiple times without creating a new
>>>> rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag
>>>> instead of using the same commit causing this issue.
>>>> >>
>>>> >> Should we create a new rc7?
>>>> >>
>>>> >> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>>> Apple, Inc
>>>> >>
>>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin
>>>> <va...@cloudera.com.INVALID> wrote:
>>>> >>>
>>>> >>> I personally find it a little weird to not have the commit in
>>>> branch-2.4.
>>>> >>>
>>>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>>> >>> (e.g. accidentally) then you lose the reference to that commit, and
>>>> >>> then the exact commit from which the rc was generated is lost.
>>>> >>>
>>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>>>> >>>>
>>>> >>>> That's weird. I see the commit but can't find it in the branch.
>>>> Was it pushed, or lost in a force push of 2.4 along the way? The change is
>>>> there, just under a different commit in the 2.4 branch.
>>>> >>>>
>>>> >>>> It doesn't necessarily invalidate the RC as it is a valid public
>>>> tagged commit and all that. I just want to be sure we do have the code from
>>>> that commit in these tatballs. It looks like it.
>>>> >>>>
>>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> Hi,
>>>> >>>>>
>>>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
>>>> branch-2.4. Release related commits I have seen so far were also part of
>>>> the branch.
>>>> >>>>>
>>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing
>>>> development version 2.4.2-SNAPSHOT" commits were simply not pushed to
>>>> spark-2.4 just the tag itself was pushed. I dont know what is the practice
>>>> in such cases but one solution is to rebase branch-2.4 changes after
>>>> 3336a21 onto these commits and do a (sorry) force push. In this case there
>>>> is no impact on this RC.
>>>> >>>>>
>>>> >>>>> Best Regards,
>>>> >>>>>
>>>> >>>>> Misi
>>>> >>>>>
>>>> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019.
>>>> márc. 8., P, 1:15):
>>>> >>>>>>
>>>> >>>>>> Please vote on releasing the following candidate as Apache Spark
>>>> version 2.4.1.
>>>> >>>>>>
>>>> >>>>>> The vote is open until March 11 PST and passes if a majority +1
>>>> PMC votes are cast, with
>>>> >>>>>> a minimum of 3 +1 votes.
>>>> >>>>>>
>>>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>> >>>>>> [ ] -1 Do not release this package because ...
>>>> >>>>>>
>>>> >>>>>> To learn more about Apache Spark, please see
>>>> http://spark.apache.org/
>>>> >>>>>>
>>>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
>>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>> >>>>>>
>>>> >>>>>> The release files, including signatures, digests, etc. can be
>>>> found at:
>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>> >>>>>>
>>>> >>>>>> Signatures used for Spark RCs can be found in this file:
>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>> >>>>>>
>>>> >>>>>> The staging repository for this release can be found at:
>>>> >>>>>>
>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>> >>>>>>
>>>> >>>>>> The documentation corresponding to this release can be found at:
>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>> >>>>>>
>>>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
>>>> following URL:
>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>> >>>>>>
>>>> >>>>>> FAQ
>>>> >>>>>>
>>>> >>>>>> =========================
>>>> >>>>>> How can I help test this release?
>>>> >>>>>> =========================
>>>> >>>>>>
>>>> >>>>>> If you are a Spark user, you can help us test this release by
>>>> taking
>>>> >>>>>> an existing Spark workload and running on this release
>>>> candidate, then
>>>> >>>>>> reporting any regressions.
>>>> >>>>>>
>>>> >>>>>> If you're working in PySpark you can set up a virtual env and
>>>> install
>>>> >>>>>> the current RC and see if anything important breaks, in the
>>>> Java/Scala
>>>> >>>>>> you can add the staging repository to your projects resolvers
>>>> and test
>>>> >>>>>> with the RC (make sure to clean up the artifact cache
>>>> before/after so
>>>> >>>>>> you don't end up building with a out of date RC going forward).
>>>> >>>>>>
>>>> >>>>>> ===========================================
>>>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>> >>>>>> ===========================================
>>>> >>>>>>
>>>> >>>>>> The current list of open tickets targeted at 2.4.1 can be found
>>>> at:
>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search for
>>>> "Target Version/s" = 2.4.1
>>>> >>>>>>
>>>> >>>>>> Committers should look at those and triage. Extremely important
>>>> bug
>>>> >>>>>> fixes, documentation, and API tweaks that impact compatibility
>>>> should
>>>> >>>>>> be worked on immediately. Everything else please retarget to an
>>>> >>>>>> appropriate release.
>>>> >>>>>>
>>>> >>>>>> ==================
>>>> >>>>>> But my bug isn't fixed?
>>>> >>>>>> ==================
>>>> >>>>>>
>>>> >>>>>> In order to make timely releases, we will typically not hold the
>>>> >>>>>> release unless the bug in question is a regression from the
>>>> previous
>>>> >>>>>> release. That being said, if there is something which is a
>>>> regression
>>>> >>>>>> that has not been correctly targeted please ping me or a
>>>> committer to
>>>> >>>>>> help target the issue.
>>>> >>>>>>
>>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution]
>>>> | Apple, Inc
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> ---------------------------------------------------------------------
>>>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >>>>>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Marcelo
>>>> >>>
>>>> >>>
>>>> ---------------------------------------------------------------------
>>>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >>>
>>>> >>
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >>
>>>> >
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
>>> --
>>> [image:
>>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>>
>> --
>> - DB Sent from my iPhone
>>
>
--
[image:
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <db...@dbtsai.com.INVALID>.
BTW, practically, is it common for users running into this bug when the
driver and executors have different object layout?
Sincerely,
DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1
On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <db...@dbtsai.com> wrote:
> Hi Xiao,
>
> I already cut rc7 and start the build process. If we definitely need this
> fix, I can cut rc8. Let me know what you think.
>
> Thanks,
>
> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <li...@databricks.com> wrote:
>
>> Hi, DB,
>>
>> Since this RC will fail, could you hold it until we fix
>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will
>> submit a PR today. The PR is small and the risk is low. This is a
>> correctness bug. It would be good to have it.
>>
>> Thanks,
>>
>> Xiao
>>
>>
>>
>>
>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid> wrote:
>>
>>> Since I can not find the commit of `Preparing development version
>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
>>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
>>> this vote fails.
>>>
>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>> Apple, Inc
>>>
>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
>>> >
>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very
>>> weird. It must be overwritten by a force push.
>>> >
>>> > DB Tsai | Siri Open Source Technologies [not a contribution] |
>>> Apple, Inc
>>> >
>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID>
>>> wrote:
>>> >>
>>> >> I was using `./do-release-docker.sh` to create a release. But since
>>> the gpg validation fails couple times when the script tried to publish the
>>> jars into Nexus, I re-ran the scripts multiple times without creating a new
>>> rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag
>>> instead of using the same commit causing this issue.
>>> >>
>>> >> Should we create a new rc7?
>>> >>
>>> >> DB Tsai | Siri Open Source Technologies [not a contribution] |
>>> Apple, Inc
>>> >>
>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin
>>> <va...@cloudera.com.INVALID> wrote:
>>> >>>
>>> >>> I personally find it a little weird to not have the commit in
>>> branch-2.4.
>>> >>>
>>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> >>> (e.g. accidentally) then you lose the reference to that commit, and
>>> >>> then the exact commit from which the rc was generated is lost.
>>> >>>
>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>>> >>>>
>>> >>>> That's weird. I see the commit but can't find it in the branch. Was
>>> it pushed, or lost in a force push of 2.4 along the way? The change is
>>> there, just under a different commit in the 2.4 branch.
>>> >>>>
>>> >>>> It doesn't necessarily invalidate the RC as it is a valid public
>>> tagged commit and all that. I just want to be sure we do have the code from
>>> that commit in these tatballs. It looks like it.
>>> >>>>
>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com>
>>> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
>>> branch-2.4. Release related commits I have seen so far were also part of
>>> the branch.
>>> >>>>>
>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing
>>> development version 2.4.2-SNAPSHOT" commits were simply not pushed to
>>> spark-2.4 just the tag itself was pushed. I dont know what is the practice
>>> in such cases but one solution is to rebase branch-2.4 changes after
>>> 3336a21 onto these commits and do a (sorry) force push. In this case there
>>> is no impact on this RC.
>>> >>>>>
>>> >>>>> Best Regards,
>>> >>>>>
>>> >>>>> Misi
>>> >>>>>
>>> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc.
>>> 8., P, 1:15):
>>> >>>>>>
>>> >>>>>> Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.1.
>>> >>>>>>
>>> >>>>>> The vote is open until March 11 PST and passes if a majority +1
>>> PMC votes are cast, with
>>> >>>>>> a minimum of 3 +1 votes.
>>> >>>>>>
>>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>> >>>>>> [ ] -1 Do not release this package because ...
>>> >>>>>>
>>> >>>>>> To learn more about Apache Spark, please see
>>> http://spark.apache.org/
>>> >>>>>>
>>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>> >>>>>>
>>> >>>>>> The release files, including signatures, digests, etc. can be
>>> found at:
>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>> >>>>>>
>>> >>>>>> Signatures used for Spark RCs can be found in this file:
>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >>>>>>
>>> >>>>>> The staging repository for this release can be found at:
>>> >>>>>>
>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>> >>>>>>
>>> >>>>>> The documentation corresponding to this release can be found at:
>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>> >>>>>>
>>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
>>> following URL:
>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>> >>>>>>
>>> >>>>>> FAQ
>>> >>>>>>
>>> >>>>>> =========================
>>> >>>>>> How can I help test this release?
>>> >>>>>> =========================
>>> >>>>>>
>>> >>>>>> If you are a Spark user, you can help us test this release by
>>> taking
>>> >>>>>> an existing Spark workload and running on this release candidate,
>>> then
>>> >>>>>> reporting any regressions.
>>> >>>>>>
>>> >>>>>> If you're working in PySpark you can set up a virtual env and
>>> install
>>> >>>>>> the current RC and see if anything important breaks, in the
>>> Java/Scala
>>> >>>>>> you can add the staging repository to your projects resolvers and
>>> test
>>> >>>>>> with the RC (make sure to clean up the artifact cache
>>> before/after so
>>> >>>>>> you don't end up building with a out of date RC going forward).
>>> >>>>>>
>>> >>>>>> ===========================================
>>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>> >>>>>> ===========================================
>>> >>>>>>
>>> >>>>>> The current list of open tickets targeted at 2.4.1 can be found
>>> at:
>>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search for
>>> "Target Version/s" = 2.4.1
>>> >>>>>>
>>> >>>>>> Committers should look at those and triage. Extremely important
>>> bug
>>> >>>>>> fixes, documentation, and API tweaks that impact compatibility
>>> should
>>> >>>>>> be worked on immediately. Everything else please retarget to an
>>> >>>>>> appropriate release.
>>> >>>>>>
>>> >>>>>> ==================
>>> >>>>>> But my bug isn't fixed?
>>> >>>>>> ==================
>>> >>>>>>
>>> >>>>>> In order to make timely releases, we will typically not hold the
>>> >>>>>> release unless the bug in question is a regression from the
>>> previous
>>> >>>>>> release. That being said, if there is something which is a
>>> regression
>>> >>>>>> that has not been correctly targeted please ping me or a
>>> committer to
>>> >>>>>> help target the issue.
>>> >>>>>>
>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution]
>>> | Apple, Inc
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> ---------------------------------------------------------------------
>>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >>>>>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Marcelo
>>> >>>
>>> >>> ---------------------------------------------------------------------
>>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >>>
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >>
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>> --
>> [image:
>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>
> --
> - DB Sent from my iPhone
>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <db...@dbtsai.com.INVALID>.
Hi Xiao,
I already cut rc7 and start the build process. If we definitely need this
fix, I can cut rc8. Let me know what you think.
Thanks,
On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <li...@databricks.com> wrote:
> Hi, DB,
>
> Since this RC will fail, could you hold it until we fix
> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will
> submit a PR today. The PR is small and the risk is low. This is a
> correctness bug. It would be good to have it.
>
> Thanks,
>
> Xiao
>
>
>
>
> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid> wrote:
>
>> Since I can not find the commit of `Preparing development version
>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
>> this vote fails.
>>
>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>> Apple, Inc
>>
>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
>> >
>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very
>> weird. It must be overwritten by a force push.
>> >
>> > DB Tsai | Siri Open Source Technologies [not a contribution] |
>> Apple, Inc
>> >
>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
>> >>
>> >> I was using `./do-release-docker.sh` to create a release. But since
>> the gpg validation fails couple times when the script tried to publish the
>> jars into Nexus, I re-ran the scripts multiple times without creating a new
>> rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag
>> instead of using the same commit causing this issue.
>> >>
>> >> Should we create a new rc7?
>> >>
>> >> DB Tsai | Siri Open Source Technologies [not a contribution] |
>> Apple, Inc
>> >>
>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin
>> <va...@cloudera.com.INVALID> wrote:
>> >>>
>> >>> I personally find it a little weird to not have the commit in
>> branch-2.4.
>> >>>
>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>> >>> (e.g. accidentally) then you lose the reference to that commit, and
>> >>> then the exact commit from which the rc was generated is lost.
>> >>>
>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>> >>>>
>> >>>> That's weird. I see the commit but can't find it in the branch. Was
>> it pushed, or lost in a force push of 2.4 along the way? The change is
>> there, just under a different commit in the 2.4 branch.
>> >>>>
>> >>>> It doesn't necessarily invalidate the RC as it is a valid public
>> tagged commit and all that. I just want to be sure we do have the code from
>> that commit in these tatballs. It looks like it.
>> >>>>
>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
>> branch-2.4. Release related commits I have seen so far were also part of
>> the branch.
>> >>>>>
>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing
>> development version 2.4.2-SNAPSHOT" commits were simply not pushed to
>> spark-2.4 just the tag itself was pushed. I dont know what is the practice
>> in such cases but one solution is to rebase branch-2.4 changes after
>> 3336a21 onto these commits and do a (sorry) force push. In this case there
>> is no impact on this RC.
>> >>>>>
>> >>>>> Best Regards,
>> >>>>>
>> >>>>> Misi
>> >>>>>
>> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc.
>> 8., P, 1:15):
>> >>>>>>
>> >>>>>> Please vote on releasing the following candidate as Apache Spark
>> version 2.4.1.
>> >>>>>>
>> >>>>>> The vote is open until March 11 PST and passes if a majority +1
>> PMC votes are cast, with
>> >>>>>> a minimum of 3 +1 votes.
>> >>>>>>
>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>> >>>>>> [ ] -1 Do not release this package because ...
>> >>>>>>
>> >>>>>> To learn more about Apache Spark, please see
>> http://spark.apache.org/
>> >>>>>>
>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>> >>>>>>
>> >>>>>> The release files, including signatures, digests, etc. can be
>> found at:
>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>> >>>>>>
>> >>>>>> Signatures used for Spark RCs can be found in this file:
>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >>>>>>
>> >>>>>> The staging repository for this release can be found at:
>> >>>>>>
>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>> >>>>>>
>> >>>>>> The documentation corresponding to this release can be found at:
>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>> >>>>>>
>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
>> following URL:
>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>> >>>>>>
>> >>>>>> FAQ
>> >>>>>>
>> >>>>>> =========================
>> >>>>>> How can I help test this release?
>> >>>>>> =========================
>> >>>>>>
>> >>>>>> If you are a Spark user, you can help us test this release by
>> taking
>> >>>>>> an existing Spark workload and running on this release candidate,
>> then
>> >>>>>> reporting any regressions.
>> >>>>>>
>> >>>>>> If you're working in PySpark you can set up a virtual env and
>> install
>> >>>>>> the current RC and see if anything important breaks, in the
>> Java/Scala
>> >>>>>> you can add the staging repository to your projects resolvers and
>> test
>> >>>>>> with the RC (make sure to clean up the artifact cache before/after
>> so
>> >>>>>> you don't end up building with a out of date RC going forward).
>> >>>>>>
>> >>>>>> ===========================================
>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>> >>>>>> ===========================================
>> >>>>>>
>> >>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search for
>> "Target Version/s" = 2.4.1
>> >>>>>>
>> >>>>>> Committers should look at those and triage. Extremely important bug
>> >>>>>> fixes, documentation, and API tweaks that impact compatibility
>> should
>> >>>>>> be worked on immediately. Everything else please retarget to an
>> >>>>>> appropriate release.
>> >>>>>>
>> >>>>>> ==================
>> >>>>>> But my bug isn't fixed?
>> >>>>>> ==================
>> >>>>>>
>> >>>>>> In order to make timely releases, we will typically not hold the
>> >>>>>> release unless the bug in question is a regression from the
>> previous
>> >>>>>> release. That being said, if there is something which is a
>> regression
>> >>>>>> that has not been correctly targeted please ping me or a committer
>> to
>> >>>>>> help target the issue.
>> >>>>>>
>> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>> Apple, Inc
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> ---------------------------------------------------------------------
>> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>>>>>
>> >>>
>> >>>
>> >>> --
>> >>> Marcelo
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
--
- DB Sent from my iPhone
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Xiao Li <li...@databricks.com>.
Hi, DB,
Since this RC will fail, could you hold it until we fix
https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will
submit a PR today. The PR is small and the risk is low. This is a
correctness bug. It would be good to have it.
Thanks,
Xiao
On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_...@apple.com.invalid> wrote:
> Since I can not find the commit of `Preparing development version
> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a
> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus,
> this vote fails.
>
> DB Tsai | Siri Open Source Technologies [not a contribution] |
> Apple, Inc
>
> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
> >
> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very
> weird. It must be overwritten by a force push.
> >
> > DB Tsai | Siri Open Source Technologies [not a contribution] |
> Apple, Inc
> >
> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
> >>
> >> I was using `./do-release-docker.sh` to create a release. But since the
> gpg validation fails couple times when the script tried to publish the jars
> into Nexus, I re-ran the scripts multiple times without creating a new rc.
> I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of
> using the same commit causing this issue.
> >>
> >> Should we create a new rc7?
> >>
> >> DB Tsai | Siri Open Source Technologies [not a contribution] |
> Apple, Inc
> >>
> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin
> <va...@cloudera.com.INVALID> wrote:
> >>>
> >>> I personally find it a little weird to not have the commit in
> branch-2.4.
> >>>
> >>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
> >>> (e.g. accidentally) then you lose the reference to that commit, and
> >>> then the exact commit from which the rc was generated is lost.
> >>>
> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
> >>>>
> >>>> That's weird. I see the commit but can't find it in the branch. Was
> it pushed, or lost in a force push of 2.4 along the way? The change is
> there, just under a different commit in the 2.4 branch.
> >>>>
> >>>> It doesn't necessarily invalidate the RC as it is a valid public
> tagged commit and all that. I just want to be sure we do have the code from
> that commit in these tatballs. It looks like it.
> >>>>
> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on
> branch-2.4. Release related commits I have seen so far were also part of
> the branch.
> >>>>>
> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing
> development version 2.4.2-SNAPSHOT" commits were simply not pushed to
> spark-2.4 just the tag itself was pushed. I dont know what is the practice
> in such cases but one solution is to rebase branch-2.4 changes after
> 3336a21 onto these commits and do a (sorry) force push. In this case there
> is no impact on this RC.
> >>>>>
> >>>>> Best Regards,
> >>>>>
> >>>>> Misi
> >>>>>
> >>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc.
> 8., P, 1:15):
> >>>>>>
> >>>>>> Please vote on releasing the following candidate as Apache Spark
> version 2.4.1.
> >>>>>>
> >>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC
> votes are cast, with
> >>>>>> a minimum of 3 +1 votes.
> >>>>>>
> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
> >>>>>> [ ] -1 Do not release this package because ...
> >>>>>>
> >>>>>> To learn more about Apache Spark, please see
> http://spark.apache.org/
> >>>>>>
> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit
> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
> >>>>>>
> >>>>>> The release files, including signatures, digests, etc. can be found
> at:
> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
> >>>>>>
> >>>>>> Signatures used for Spark RCs can be found in this file:
> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
> >>>>>>
> >>>>>> The staging repository for this release can be found at:
> >>>>>>
> https://repository.apache.org/content/repositories/orgapachespark-1308/
> >>>>>>
> >>>>>> The documentation corresponding to this release can be found at:
> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
> >>>>>>
> >>>>>> The list of bug fixes going into 2.4.1 can be found at the
> following URL:
> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
> >>>>>>
> >>>>>> FAQ
> >>>>>>
> >>>>>> =========================
> >>>>>> How can I help test this release?
> >>>>>> =========================
> >>>>>>
> >>>>>> If you are a Spark user, you can help us test this release by taking
> >>>>>> an existing Spark workload and running on this release candidate,
> then
> >>>>>> reporting any regressions.
> >>>>>>
> >>>>>> If you're working in PySpark you can set up a virtual env and
> install
> >>>>>> the current RC and see if anything important breaks, in the
> Java/Scala
> >>>>>> you can add the staging repository to your projects resolvers and
> test
> >>>>>> with the RC (make sure to clean up the artifact cache before/after
> so
> >>>>>> you don't end up building with a out of date RC going forward).
> >>>>>>
> >>>>>> ===========================================
> >>>>>> What should happen to JIRA tickets still targeting 2.4.1?
> >>>>>> ===========================================
> >>>>>>
> >>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
> >>>>>> https://issues.apache.org/jira/projects/SPARK and search for
> "Target Version/s" = 2.4.1
> >>>>>>
> >>>>>> Committers should look at those and triage. Extremely important bug
> >>>>>> fixes, documentation, and API tweaks that impact compatibility
> should
> >>>>>> be worked on immediately. Everything else please retarget to an
> >>>>>> appropriate release.
> >>>>>>
> >>>>>> ==================
> >>>>>> But my bug isn't fixed?
> >>>>>> ==================
> >>>>>>
> >>>>>> In order to make timely releases, we will typically not hold the
> >>>>>> release unless the bug in question is a regression from the previous
> >>>>>> release. That being said, if there is something which is a
> regression
> >>>>>> that has not been correctly targeted please ping me or a committer
> to
> >>>>>> help target the issue.
> >>>>>>
> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] |
> Apple, Inc
> >>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>>>>
> >>>
> >>>
> >>> --
> >>> Marcelo
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
[image:
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <d_...@apple.com.INVALID>.
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.
DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> On Mar 8, 2019, at 11:45 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <va...@cloudera.com.INVALID> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <d_...@apple.com.INVALID>.
Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_...@apple.com.INVALID> wrote:
>
> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>
> Should we create a new rc7?
>
> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>
>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <va...@cloudera.com.INVALID> wrote:
>>
>> I personally find it a little weird to not have the commit in branch-2.4.
>>
>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>> (e.g. accidentally) then you lose the reference to that commit, and
>> then the exact commit from which the rc was generated is lost.
>>
>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>>>
>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>
>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>
>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>
>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>
>>>> Best Regards,
>>>>
>>>> Misi
>>>>
>>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>
>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>
>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>> a minimum of 3 +1 votes.
>>>>>
>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>> [ ] -1 Do not release this package because ...
>>>>>
>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>
>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>
>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>
>>>>> Signatures used for Spark RCs can be found in this file:
>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>
>>>>> The staging repository for this release can be found at:
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>
>>>>> The documentation corresponding to this release can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>
>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>
>>>>> FAQ
>>>>>
>>>>> =========================
>>>>> How can I help test this release?
>>>>> =========================
>>>>>
>>>>> If you are a Spark user, you can help us test this release by taking
>>>>> an existing Spark workload and running on this release candidate, then
>>>>> reporting any regressions.
>>>>>
>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>> you can add the staging repository to your projects resolvers and test
>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>> you don't end up building with a out of date RC going forward).
>>>>>
>>>>> ===========================================
>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>> ===========================================
>>>>>
>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>
>>>>> Committers should look at those and triage. Extremely important bug
>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>> be worked on immediately. Everything else please retarget to an
>>>>> appropriate release.
>>>>>
>>>>> ==================
>>>>> But my bug isn't fixed?
>>>>> ==================
>>>>>
>>>>> In order to make timely releases, we will typically not hold the
>>>>> release unless the bug in question is a regression from the previous
>>>>> release. That being said, if there is something which is a regression
>>>>> that has not been correctly targeted please ping me or a committer to
>>>>> help target the issue.
>>>>>
>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by DB Tsai <d_...@apple.com.INVALID>.
I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
Should we create a new rc7?
DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <va...@cloudera.com.INVALID> wrote:
>
> I personally find it a little weird to not have the commit in branch-2.4.
>
> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
> (e.g. accidentally) then you lose the reference to that commit, and
> then the exact commit from which the rc was generated is lost.
>
> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>>
>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>
>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>
>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>
>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>
>>> Best Regards,
>>>
>>> Misi
>>>
>>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>
>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>
>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>> a minimum of 3 +1 votes.
>>>>
>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>> [ ] -1 Do not release this package because ...
>>>>
>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>
>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>
>>>> The release files, including signatures, digests, etc. can be found at:
>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>
>>>> Signatures used for Spark RCs can be found in this file:
>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>
>>>> The staging repository for this release can be found at:
>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>
>>>> The documentation corresponding to this release can be found at:
>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>
>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>
>>>> FAQ
>>>>
>>>> =========================
>>>> How can I help test this release?
>>>> =========================
>>>>
>>>> If you are a Spark user, you can help us test this release by taking
>>>> an existing Spark workload and running on this release candidate, then
>>>> reporting any regressions.
>>>>
>>>> If you're working in PySpark you can set up a virtual env and install
>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>> you can add the staging repository to your projects resolvers and test
>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>> you don't end up building with a out of date RC going forward).
>>>>
>>>> ===========================================
>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>> ===========================================
>>>>
>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>
>>>> Committers should look at those and triage. Extremely important bug
>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>> be worked on immediately. Everything else please retarget to an
>>>> appropriate release.
>>>>
>>>> ==================
>>>> But my bug isn't fixed?
>>>> ==================
>>>>
>>>> In order to make timely releases, we will typically not hold the
>>>> release unless the bug in question is a regression from the previous
>>>> release. That being said, if there is something which is a regression
>>>> that has not been correctly targeted please ping me or a committer to
>>>> help target the issue.
>>>>
>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Marcelo Vanzin <va...@cloudera.com.INVALID>.
I personally find it a little weird to not have the commit in branch-2.4.
Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
(e.g. accidentally) then you lose the reference to that commit, and
then the exact commit from which the rc was generated is lost.
On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sr...@gmail.com> wrote:
>
> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>
> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>
> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>
>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>
>> Best Regards,
>>
>> Misi
>>
>> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>
>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>
>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>> a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>
>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>
>>> FAQ
>>>
>>> =========================
>>> How can I help test this release?
>>> =========================
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===========================================
>>> What should happen to JIRA tickets still targeting 2.4.1?
>>> ===========================================
>>>
>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==================
>>> But my bug isn't fixed?
>>> ==================
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Sean Owen <sr...@gmail.com>.
That's weird. I see the commit but can't find it in the branch. Was it
pushed, or lost in a force push of 2.4 along the way? The change is there,
just under a different commit in the 2.4 branch.
It doesn't necessarily invalidate the RC as it is a valid public tagged
commit and all that. I just want to be sure we do have the code from that
commit in these tatballs. It looks like it.
On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <mi...@gmail.com> wrote:
> Hi,
>
> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4.
> Release related commits I have seen so far were also part of the branch.
>
> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing
> development version 2.4.2-SNAPSHOT" commits were simply not pushed to
> spark-2.4 just the tag itself was pushed. I dont know what is the practice
> in such cases but one solution is to rebase branch-2.4 changes
> after 3336a21 onto these commits and do a (sorry) force push. In this case
> there is no impact on this RC.
>
> Best Regards,
>
> Misi
>
> DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P,
> 1:15):
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.4.1.
>>
>> The vote is open until March 11 PST and passes if a majority +1 PMC votes
>> are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.1
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.1-rc6 (commit
>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>
>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>
>> FAQ
>>
>> =========================
>> How can I help test this release?
>> =========================
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===========================================
>> What should happen to JIRA tickets still targeting 2.4.1?
>> ===========================================
>>
>> The current list of open tickets targeted at 2.4.1 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 2.4.1
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==================
>> But my bug isn't fixed?
>> ==================
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>> DB Tsai | Siri Open Source Technologies [not a contribution] |
>> Apple, Inc
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [VOTE] Release Apache Spark 2.4.1 (RC6)
Posted by Mihály Tóth <mi...@gmail.com>.
Hi,
I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4.
Release related commits I have seen so far were also part of the branch.
I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development
version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just
the tag itself was pushed. I dont know what is the practice in such cases
but one solution is to rebase branch-2.4 changes after 3336a21 onto these
commits and do a (sorry) force push. In this case there is no impact on
this RC.
Best Regards,
Misi
DB Tsai <d_...@apple.com.invalid> ezt írta (időpont: 2019. márc. 8., P,
1:15):
> Please vote on releasing the following candidate as Apache Spark version
> 2.4.1.
>
> The vote is open until March 11 PST and passes if a majority +1 PMC votes
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.1
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.1-rc6 (commit
> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
> https://github.com/apache/spark/tree/v2.4.1-rc6
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1308/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>
> The list of bug fixes going into 2.4.1 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.1?
> ===========================================
>
> The current list of open tickets targeted at 2.4.1 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.1
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
> DB Tsai | Siri Open Source Technologies [not a contribution] |
> Apple, Inc
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>