You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Holden Karau <ho...@pigscanfly.ca> on 2020/02/04 21:44:45 UTC

Initial Decom PR for Spark 3?

Hi Y’all,

I’ve got a K8s graceful decom PR (
https://github.com/apache/spark/pull/26440
 ) I’d love to try and get in for Spark 3, but I don’t want to push on it
if folks don’t think it’s worth it. I’ve been working on it since 2017 and
it was really close in November but then I had the crash and had to step
back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for
awhile so those points are in its favour. It does however change things in
core which is not great.

Cheers,

Holden
-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Initial Decom PR for Spark 3?

Posted by Holden Karau <ho...@pigscanfly.ca>.
For follow up while I've backported this in some internal releases I'm not
considering a candidate for backporting to Spark 3 anymore. I should have
updated the thread with that. The design doc is linked in the PR.

On Thu, Jun 18, 2020 at 6:05 PM Hyukjin Kwon <gu...@gmail.com> wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>> and it was really close in November but then I had the crash and had to
>>> step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>> for awhile so those points are in its favour. It does however change things
>>> in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Initial Decom PR for Spark 3?

Posted by Hyukjin Kwon <gu...@gmail.com>.
See https://spark.apache.org/improvement-proposals.html

On Tue, 23 Jun 2020, 07:01 Stephen Boesch, <ja...@gmail.com> wrote:

> I guess I missed that "community decision" where the existing design
> document that had been reviewed was put aside and a new SPIP  document was
> required.
>
> On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Yeah, I believe the community decided to do a SPIP for such significant
>> changes. It would be best if we stick to the standard approaches.
>>
>> 2020년 6월 21일 (일) 오전 8:52, Holden Karau <ho...@pigscanfly.ca>님이 작성:
>>
>>> I believe so, however since Hyukjin is a committer and has asked for an
>>> SPIP I'll be making an SPIP for this next week. I hope to send out the
>>> draft for comment by the end of Spark summit. I'll be using the same design
>>> document for the design component, so if anyone has input on the design
>>> document feel free to start leaving comments there now.
>>>
>>> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <ja...@gmail.com>
>>> wrote:
>>>
>>>> Hi given there is a design doc (contrary to that common) - is this
>>>> going to move forward?
>>>>
>>>> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <gu...@gmail.com> wrote:
>>>>
>>>>> Looks it had to be with SPIP and a proper design doc to discuss.
>>>>>
>>>>> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:
>>>>>
>>>>>> I'd be willing to pull this in, unless others have concerns post
>>>>>> branch-cut.
>>>>>>
>>>>>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Y’all,
>>>>>>>
>>>>>>> I’ve got a K8s graceful decom PR (
>>>>>>> https://github.com/apache/spark/pull/26440
>>>>>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push
>>>>>>> on it if folks don’t think it’s worth it. I’ve been working on it since
>>>>>>> 2017 and it was really close in November but then I had the crash and had
>>>>>>> to step back for awhile.
>>>>>>>
>>>>>>> It’s effectiveness is behind a feature flag and it’s been
>>>>>>> outstanding for awhile so those points are in its favour. It does however
>>>>>>> change things in core which is not great.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Holden
>>>>>>> --
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>>
>>>>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Initial Decom PR for Spark 3?

Posted by Stephen Boesch <ja...@gmail.com>.
I guess I missed that "community decision" where the existing design
document that had been reviewed was put aside and a new SPIP  document was
required.

On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon <gu...@gmail.com> wrote:

> Yeah, I believe the community decided to do a SPIP for such significant
> changes. It would be best if we stick to the standard approaches.
>
> 2020년 6월 21일 (일) 오전 8:52, Holden Karau <ho...@pigscanfly.ca>님이 작성:
>
>> I believe so, however since Hyukjin is a committer and has asked for an
>> SPIP I'll be making an SPIP for this next week. I hope to send out the
>> draft for comment by the end of Spark summit. I'll be using the same design
>> document for the design component, so if anyone has input on the design
>> document feel free to start leaving comments there now.
>>
>> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <ja...@gmail.com> wrote:
>>
>>> Hi given there is a design doc (contrary to that common) - is this going
>>> to move forward?
>>>
>>> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <gu...@gmail.com> wrote:
>>>
>>>> Looks it had to be with SPIP and a proper design doc to discuss.
>>>>
>>>> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:
>>>>
>>>>> I'd be willing to pull this in, unless others have concerns post
>>>>> branch-cut.
>>>>>
>>>>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca>
>>>>> wrote:
>>>>>
>>>>>> Hi Y’all,
>>>>>>
>>>>>> I’ve got a K8s graceful decom PR (
>>>>>> https://github.com/apache/spark/pull/26440
>>>>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push
>>>>>> on it if folks don’t think it’s worth it. I’ve been working on it since
>>>>>> 2017 and it was really close in November but then I had the crash and had
>>>>>> to step back for awhile.
>>>>>>
>>>>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>>>>> for awhile so those points are in its favour. It does however change things
>>>>>> in core which is not great.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Holden
>>>>>> --
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>>
>>>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Initial Decom PR for Spark 3?

Posted by Hyukjin Kwon <gu...@gmail.com>.
Yeah, I believe the community decided to do a SPIP for such significant
changes. It would be best if we stick to the standard approaches.

2020년 6월 21일 (일) 오전 8:52, Holden Karau <ho...@pigscanfly.ca>님이 작성:

> I believe so, however since Hyukjin is a committer and has asked for an
> SPIP I'll be making an SPIP for this next week. I hope to send out the
> draft for comment by the end of Spark summit. I'll be using the same design
> document for the design component, so if anyone has input on the design
> document feel free to start leaving comments there now.
>
> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <ja...@gmail.com> wrote:
>
>> Hi given there is a design doc (contrary to that common) - is this going
>> to move forward?
>>
>> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> Looks it had to be with SPIP and a proper design doc to discuss.
>>>
>>> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:
>>>
>>>> I'd be willing to pull this in, unless others have concerns post
>>>> branch-cut.
>>>>
>>>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca>
>>>> wrote:
>>>>
>>>>> Hi Y’all,
>>>>>
>>>>> I’ve got a K8s graceful decom PR (
>>>>> https://github.com/apache/spark/pull/26440
>>>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>>>> and it was really close in November but then I had the crash and had to
>>>>> step back for awhile.
>>>>>
>>>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>>>> for awhile so those points are in its favour. It does however change things
>>>>> in core which is not great.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Holden
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>
>>>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Initial Decom PR for Spark 3?

Posted by Holden Karau <ho...@pigscanfly.ca>.
I believe so, however since Hyukjin is a committer and has asked for an
SPIP I'll be making an SPIP for this next week. I hope to send out the
draft for comment by the end of Spark summit. I'll be using the same design
document for the design component, so if anyone has input on the design
document feel free to start leaving comments there now.

On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <ja...@gmail.com> wrote:

> Hi given there is a design doc (contrary to that common) - is this going
> to move forward?
>
> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Looks it had to be with SPIP and a proper design doc to discuss.
>>
>> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:
>>
>>> I'd be willing to pull this in, unless others have concerns post
>>> branch-cut.
>>>
>>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca>
>>> wrote:
>>>
>>>> Hi Y’all,
>>>>
>>>> I’ve got a K8s graceful decom PR (
>>>> https://github.com/apache/spark/pull/26440
>>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>>> and it was really close in November but then I had the crash and had to
>>>> step back for awhile.
>>>>
>>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>>> for awhile so those points are in its favour. It does however change things
>>>> in core which is not great.
>>>>
>>>> Cheers,
>>>>
>>>> Holden
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Initial Decom PR for Spark 3?

Posted by Stephen Boesch <ja...@gmail.com>.
Hi given there is a design doc (contrary to that common) - is this going to
move forward?

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <gu...@gmail.com> wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>> and it was really close in November but then I had the crash and had to
>>> step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>> for awhile so those points are in its favour. It does however change things
>>> in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Initial Decom PR for Spark 3?

Posted by Stephen Boesch <ja...@gmail.com>.
Second paragraph of the PR lists the design doc.

> There is a design document at
https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <gu...@gmail.com> wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on
>>> it if folks don’t think it’s worth it. I’ve been working on it since 2017
>>> and it was really close in November but then I had the crash and had to
>>> step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been outstanding
>>> for awhile so those points are in its favour. It does however change things
>>> in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Initial Decom PR for Spark 3?

Posted by Hyukjin Kwon <gu...@gmail.com>.
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <ee...@redhat.com>님이 작성:

> I'd be willing to pull this in, unless others have concerns post
> branch-cut.
>
> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> Hi Y’all,
>>
>> I’ve got a K8s graceful decom PR (
>> https://github.com/apache/spark/pull/26440
>>  ) I’d love to try and get in for Spark 3, but I don’t want to push on it
>> if folks don’t think it’s worth it. I’ve been working on it since 2017 and
>> it was really close in November but then I had the crash and had to step
>> back for awhile.
>>
>> It’s effectiveness is behind a feature flag and it’s been outstanding for
>> awhile so those points are in its favour. It does however change things in
>> core which is not great.
>>
>> Cheers,
>>
>> Holden
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Initial Decom PR for Spark 3?

Posted by Erik Erlandson <ee...@redhat.com>.
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <ho...@pigscanfly.ca> wrote:

> Hi Y’all,
>
> I’ve got a K8s graceful decom PR (
> https://github.com/apache/spark/pull/26440
>  ) I’d love to try and get in for Spark 3, but I don’t want to push on it
> if folks don’t think it’s worth it. I’ve been working on it since 2017 and
> it was really close in November but then I had the crash and had to step
> back for awhile.
>
> It’s effectiveness is behind a feature flag and it’s been outstanding for
> awhile so those points are in its favour. It does however change things in
> core which is not great.
>
> Cheers,
>
> Holden
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>