You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Takeshi Yamamuro <li...@gmail.com> on 2020/08/03 08:37:38 UTC
spark-on-k8s is still experimental?
Hi, all
A Spark user I know asked me this question.
I checked the the spark-on-k8s document and it says;
```
**The Kubernetes scheduler is currently experimental.
In future versions, there may be behavioral changes around configuration,
container images and entrypoints.**
```
https://github.com/apache/spark/blame/master/docs/running-on-kubernetes.md#L26-L28
This statement was added when v2.3.0 released (
https://issues.apache.org/jira/browse/SPARK-23104).
Does this mean that the v2.3.0 and v3.0.0 releases have the same
development stage?
I'm a bit worried that users read the statement in the doc and give up
starting this feature.
(In fact, the user who asked me seemed to think so)
I'm not familiar with that part, so does anyone know
when this statement can be removed from the doc?
Any milestone for that?
Thanks in advance,
Takeshi
--
---
Takeshi Yamamuro
Re: spark-on-k8s is still experimental?
Posted by Prashant Sharma <sc...@gmail.com>.
My thoughts are, External shuffle service is not a blocker for spark on k8s
to be a production ready.
Others may think otherwise, but there are other ways to enable auto
scaling. External shuffle service feature will be useful for all the ways
of deployments be it yarn, standalone, k8s and not just k8s.
About the GA, I have not yet seen a very large deployment working yet.
Others can share, how they are using spark on k8s, that can give us more
confidence on moving towards GA.
Thanks,
On Thu, Aug 6, 2020 at 9:18 AM Holden Karau <ho...@pigscanfly.ca> wrote:
> Sounds good. I think we can make a slightly stronger statement than that
> one (left a comment, but it's my own thoughts so others should chime in if
> they have a different opinion).
>
> On Wed, Aug 5, 2020 at 7:32 PM Takeshi Yamamuro <li...@gmail.com>
> wrote:
>
>> Thanks for the info, all. okay, I understood that we need more time to
>> announce GA officially.
>> But, I'm still worried that users hesitate a bit to use this feature by
>> referring to the statement in the doc,
>> so how about updating it according to the current situation? Please check
>> my suggestion in https://github.com/apache/spark/pull/29368.
>>
>> Anyway, many thanks!
>>
>>
>> On Tue, Aug 4, 2020 at 12:26 AM Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>>> There was discussion around removing the statement and declaring it GA
>>> but I believe it was decided to leave it in until an external shuffle
>>> service is supported on K8s.
>>>
>>> On Mon, Aug 3, 2020 at 2:45 AM JackyLee <qc...@163.com> wrote:
>>>
>>>> +1. It has been worked well in our company and we has used it to support
>>>> online services since March in this year.
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
Re: spark-on-k8s is still experimental?
Posted by Holden Karau <ho...@pigscanfly.ca>.
Sounds good. I think we can make a slightly stronger statement than that
one (left a comment, but it's my own thoughts so others should chime in if
they have a different opinion).
On Wed, Aug 5, 2020 at 7:32 PM Takeshi Yamamuro <li...@gmail.com>
wrote:
> Thanks for the info, all. okay, I understood that we need more time to
> announce GA officially.
> But, I'm still worried that users hesitate a bit to use this feature by
> referring to the statement in the doc,
> so how about updating it according to the current situation? Please check
> my suggestion in https://github.com/apache/spark/pull/29368.
>
> Anyway, many thanks!
>
>
> On Tue, Aug 4, 2020 at 12:26 AM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> There was discussion around removing the statement and declaring it GA
>> but I believe it was decided to leave it in until an external shuffle
>> service is supported on K8s.
>>
>> On Mon, Aug 3, 2020 at 2:45 AM JackyLee <qc...@163.com> wrote:
>>
>>> +1. It has been worked well in our company and we has used it to support
>>> online services since March in this year.
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>
> --
> ---
> Takeshi Yamamuro
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: spark-on-k8s is still experimental?
Posted by Takeshi Yamamuro <li...@gmail.com>.
Thanks for the info, all. okay, I understood that we need more time to
announce GA officially.
But, I'm still worried that users hesitate a bit to use this feature by
referring to the statement in the doc,
so how about updating it according to the current situation? Please check
my suggestion in https://github.com/apache/spark/pull/29368.
Anyway, many thanks!
On Tue, Aug 4, 2020 at 12:26 AM Holden Karau <ho...@pigscanfly.ca> wrote:
> There was discussion around removing the statement and declaring it GA but
> I believe it was decided to leave it in until an external shuffle service
> is supported on K8s.
>
> On Mon, Aug 3, 2020 at 2:45 AM JackyLee <qc...@163.com> wrote:
>
>> +1. It has been worked well in our company and we has used it to support
>> online services since March in this year.
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
--
---
Takeshi Yamamuro
Re: spark-on-k8s is still experimental?
Posted by Holden Karau <ho...@pigscanfly.ca>.
There was discussion around removing the statement and declaring it GA but
I believe it was decided to leave it in until an external shuffle service
is supported on K8s.
On Mon, Aug 3, 2020 at 2:45 AM JackyLee <qc...@163.com> wrote:
> +1. It has been worked well in our company and we has used it to support
> online services since March in this year.
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: spark-on-k8s is still experimental?
Posted by JackyLee <qc...@163.com>.
+1. It has been worked well in our company and we has used it to support
online services since March in this year.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: spark-on-k8s is still experimental?
Posted by Sean Owen <sr...@gmail.com>.
Likewise, I'm not super familiar with this integration, but, it's been
out for several minor and one new major version, and doubt that it is
any more 'experimental' now that it ever will be.
Unless someone who knows more suggests that it's still really a WIP,
or that some aspects are still fairly subject to change, I think we
could remove those statements.
On Mon, Aug 3, 2020 at 3:38 AM Takeshi Yamamuro <li...@gmail.com> wrote:
>
> Hi, all
>
> A Spark user I know asked me this question.
> I checked the the spark-on-k8s document and it says;
> ```
> **The Kubernetes scheduler is currently experimental.
> In future versions, there may be behavioral changes around configuration,
> container images and entrypoints.**
> ```
> https://github.com/apache/spark/blame/master/docs/running-on-kubernetes.md#L26-L28
>
> This statement was added when v2.3.0 released (https://issues.apache.org/jira/browse/SPARK-23104).
> Does this mean that the v2.3.0 and v3.0.0 releases have the same development stage?
> I'm a bit worried that users read the statement in the doc and give up starting this feature.
> (In fact, the user who asked me seemed to think so)
>
> I'm not familiar with that part, so does anyone know
> when this statement can be removed from the doc?
> Any milestone for that?
>
> Thanks in advance,
> Takeshi
>
> --
> ---
> Takeshi Yamamuro
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org