You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Pralabh Kumar <pr...@gmail.com> on 2022/02/04 10:20:58 UTC

Spark on K8s : property simillar to yarn.max.application.attempt

Hi Spark Team

I am running spark on K8s and looking for a
property/mechanism similar to  yarn.max.application.attempt . I know this
is not really a spark question , but i thought if anyone have faced the
similar issue,

Basically I want if my driver pod fails , it should be retried on a
different machine . Is there a way to do the same .

Regards
Pralabh Kumar

RE: Spark on K8s : property simillar to yarn.max.application.attempt

Posted by "Agarwal, Janak" <ja...@amazon.com.INVALID>.
Agree with Mich.

You can restart the full job, which means it is almost like submitted the same job again. The driver state is not preserved. In this case, you will have to make sure the job is idempotent.

From: Mich Talebzadeh <mi...@gmail.com>
Sent: Friday, February 4, 2022 11:07 AM
To: Pablo Langa Blanco <so...@gmail.com>
Cc: Pralabh Kumar <pr...@gmail.com>; dev <de...@spark.apache.org>; user.spark <us...@spark.apache.org>
Subject: RE: [EXTERNAL] Spark on K8s : property simillar to yarn.max.application.attempt


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.


Sure that may well be. However, we have to remember that even if the driver pod is resurrected, the driver is the one that instructs the Kube apiserver to schedule the executors. Additionally, in Spark itself when the driver node dies then the whole spark job dies. So the question is how the state is going to be preserved? With replacing the executors there ought to be no issues but about the driver pod I am not sure and it is something that needs to be clarified.

[image.png]

cheers


 [https://docs.google.com/uc?export=download&id=1-q7RFGRfLMObPuQPWSd9sl_H1UPNFaIZ&revid=0B1BiUVX33unjMWtVUWpINWFCd0ZQTlhTRHpGckh4Wlg4RG80PQ]   view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.




On Fri, 4 Feb 2022 at 16:17, Pablo Langa Blanco <so...@gmail.com>> wrote:
Hi Pralabh,

I understand that this is a question more related to K8s than to Spark itself. In K8s you have a set of objects to control the state of your pods, which allow you, among other things, to redeploy pods that have failed. This would apply to the pod containing the driver container.
https://kubernetes.io/docs/concepts/workloads/

I hope it helps you!

Regards

On Fri, Feb 4, 2022 at 7:31 AM Mich Talebzadeh <mi...@gmail.com>> wrote:
Not as far as I know. If your driver pod fails, then you need to resubmit the job. I cannot see what else can be done?



HTH


 [https://docs.google.com/uc?export=download&id=1-q7RFGRfLMObPuQPWSd9sl_H1UPNFaIZ&revid=0B1BiUVX33unjMWtVUWpINWFCd0ZQTlhTRHpGckh4Wlg4RG80PQ]   view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.




On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar <pr...@gmail.com>> wrote:
Hi Spark Team

I am running spark on K8s and looking for a property/mechanism similar to  yarn.max.application.attempt . I know this is not really a spark question , but i thought if anyone have faced the similar issue,

Basically I want if my driver pod fails , it should be retried on a different machine . Is there a way to do the same .

Regards
Pralabh Kumar

RE: Spark on K8s : property simillar to yarn.max.application.attempt

Posted by "Agarwal, Janak" <ja...@amazon.com.INVALID>.
Agree with Mich.

You can restart the full job, which means it is almost like submitted the same job again. The driver state is not preserved. In this case, you will have to make sure the job is idempotent.

From: Mich Talebzadeh <mi...@gmail.com>
Sent: Friday, February 4, 2022 11:07 AM
To: Pablo Langa Blanco <so...@gmail.com>
Cc: Pralabh Kumar <pr...@gmail.com>; dev <de...@spark.apache.org>; user.spark <us...@spark.apache.org>
Subject: RE: [EXTERNAL] Spark on K8s : property simillar to yarn.max.application.attempt


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.


Sure that may well be. However, we have to remember that even if the driver pod is resurrected, the driver is the one that instructs the Kube apiserver to schedule the executors. Additionally, in Spark itself when the driver node dies then the whole spark job dies. So the question is how the state is going to be preserved? With replacing the executors there ought to be no issues but about the driver pod I am not sure and it is something that needs to be clarified.

[image.png]

cheers


 [https://docs.google.com/uc?export=download&id=1-q7RFGRfLMObPuQPWSd9sl_H1UPNFaIZ&revid=0B1BiUVX33unjMWtVUWpINWFCd0ZQTlhTRHpGckh4Wlg4RG80PQ]   view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.




On Fri, 4 Feb 2022 at 16:17, Pablo Langa Blanco <so...@gmail.com>> wrote:
Hi Pralabh,

I understand that this is a question more related to K8s than to Spark itself. In K8s you have a set of objects to control the state of your pods, which allow you, among other things, to redeploy pods that have failed. This would apply to the pod containing the driver container.
https://kubernetes.io/docs/concepts/workloads/

I hope it helps you!

Regards

On Fri, Feb 4, 2022 at 7:31 AM Mich Talebzadeh <mi...@gmail.com>> wrote:
Not as far as I know. If your driver pod fails, then you need to resubmit the job. I cannot see what else can be done?



HTH


 [https://docs.google.com/uc?export=download&id=1-q7RFGRfLMObPuQPWSd9sl_H1UPNFaIZ&revid=0B1BiUVX33unjMWtVUWpINWFCd0ZQTlhTRHpGckh4Wlg4RG80PQ]   view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.




On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar <pr...@gmail.com>> wrote:
Hi Spark Team

I am running spark on K8s and looking for a property/mechanism similar to  yarn.max.application.attempt . I know this is not really a spark question , but i thought if anyone have faced the similar issue,

Basically I want if my driver pod fails , it should be retried on a different machine . Is there a way to do the same .

Regards
Pralabh Kumar

Re: Spark on K8s : property simillar to yarn.max.application.attempt

Posted by Mich Talebzadeh <mi...@gmail.com>.
Sure that may well be. However, we have to remember that even if the driver
pod is resurrected, the driver is the one that instructs the Kube apiserver
to schedule the executors. Additionally, in Spark itself when the driver
node dies then the whole spark job dies. So the question is how the state
is going to be preserved? With replacing the executors there ought to be no
issues but about the driver pod I am not sure and it is something that
needs to be clarified.

[image: image.png]

cheers



   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 4 Feb 2022 at 16:17, Pablo Langa Blanco <so...@gmail.com> wrote:

> Hi Pralabh,
>
> I understand that this is a question more related to K8s than to Spark
> itself. In K8s you have a set of objects to control the state of your pods,
> which allow you, among other things, to redeploy pods that have failed.
> This would apply to the pod containing the driver container.
> https://kubernetes.io/docs/concepts/workloads/
>
> I hope it helps you!
>
> Regards
>
> On Fri, Feb 4, 2022 at 7:31 AM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Not as far as I know. If your driver pod fails, then you need to resubmit
>> the job. I cannot see what else can be done?
>>
>>
>> HTH
>>
>>
>>    view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar <pr...@gmail.com>
>> wrote:
>>
>>> Hi Spark Team
>>>
>>> I am running spark on K8s and looking for a
>>> property/mechanism similar to  yarn.max.application.attempt . I know this
>>> is not really a spark question , but i thought if anyone have faced the
>>> similar issue,
>>>
>>> Basically I want if my driver pod fails , it should be retried on a
>>> different machine . Is there a way to do the same .
>>>
>>> Regards
>>> Pralabh Kumar
>>>
>>

Re: Spark on K8s : property simillar to yarn.max.application.attempt

Posted by Mich Talebzadeh <mi...@gmail.com>.
Sure that may well be. However, we have to remember that even if the driver
pod is resurrected, the driver is the one that instructs the Kube apiserver
to schedule the executors. Additionally, in Spark itself when the driver
node dies then the whole spark job dies. So the question is how the state
is going to be preserved? With replacing the executors there ought to be no
issues but about the driver pod I am not sure and it is something that
needs to be clarified.

[image: image.png]

cheers



   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 4 Feb 2022 at 16:17, Pablo Langa Blanco <so...@gmail.com> wrote:

> Hi Pralabh,
>
> I understand that this is a question more related to K8s than to Spark
> itself. In K8s you have a set of objects to control the state of your pods,
> which allow you, among other things, to redeploy pods that have failed.
> This would apply to the pod containing the driver container.
> https://kubernetes.io/docs/concepts/workloads/
>
> I hope it helps you!
>
> Regards
>
> On Fri, Feb 4, 2022 at 7:31 AM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Not as far as I know. If your driver pod fails, then you need to resubmit
>> the job. I cannot see what else can be done?
>>
>>
>> HTH
>>
>>
>>    view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar <pr...@gmail.com>
>> wrote:
>>
>>> Hi Spark Team
>>>
>>> I am running spark on K8s and looking for a
>>> property/mechanism similar to  yarn.max.application.attempt . I know this
>>> is not really a spark question , but i thought if anyone have faced the
>>> similar issue,
>>>
>>> Basically I want if my driver pod fails , it should be retried on a
>>> different machine . Is there a way to do the same .
>>>
>>> Regards
>>> Pralabh Kumar
>>>
>>

Re: Spark on K8s : property simillar to yarn.max.application.attempt

Posted by Pablo Langa Blanco <so...@gmail.com>.
Hi Pralabh,

I understand that this is a question more related to K8s than to Spark
itself. In K8s you have a set of objects to control the state of your pods,
which allow you, among other things, to redeploy pods that have failed.
This would apply to the pod containing the driver container.
https://kubernetes.io/docs/concepts/workloads/

I hope it helps you!

Regards

On Fri, Feb 4, 2022 at 7:31 AM Mich Talebzadeh <mi...@gmail.com>
wrote:

> Not as far as I know. If your driver pod fails, then you need to resubmit
> the job. I cannot see what else can be done?
>
>
> HTH
>
>
>    view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar <pr...@gmail.com> wrote:
>
>> Hi Spark Team
>>
>> I am running spark on K8s and looking for a
>> property/mechanism similar to  yarn.max.application.attempt . I know this
>> is not really a spark question , but i thought if anyone have faced the
>> similar issue,
>>
>> Basically I want if my driver pod fails , it should be retried on a
>> different machine . Is there a way to do the same .
>>
>> Regards
>> Pralabh Kumar
>>
>

Re: Spark on K8s : property simillar to yarn.max.application.attempt

Posted by Mich Talebzadeh <mi...@gmail.com>.
Not as far as I know. If your driver pod fails, then you need to resubmit
the job. I cannot see what else can be done?


HTH


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar <pr...@gmail.com> wrote:

> Hi Spark Team
>
> I am running spark on K8s and looking for a
> property/mechanism similar to  yarn.max.application.attempt . I know this
> is not really a spark question , but i thought if anyone have faced the
> similar issue,
>
> Basically I want if my driver pod fails , it should be retried on a
> different machine . Is there a way to do the same .
>
> Regards
> Pralabh Kumar
>

Re: Spark on K8s : property simillar to yarn.max.application.attempt

Posted by Mich Talebzadeh <mi...@gmail.com>.
Not as far as I know. If your driver pod fails, then you need to resubmit
the job. I cannot see what else can be done?


HTH


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar <pr...@gmail.com> wrote:

> Hi Spark Team
>
> I am running spark on K8s and looking for a
> property/mechanism similar to  yarn.max.application.attempt . I know this
> is not really a spark question , but i thought if anyone have faced the
> similar issue,
>
> Basically I want if my driver pod fails , it should be retried on a
> different machine . Is there a way to do the same .
>
> Regards
> Pralabh Kumar
>