You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by unk1102 <um...@gmail.com> on 2016/02/25 17:54:03 UTC
Spark 1.6.0 running jobs in yarn shows negative no of tasks in
executor
Hi I have spark job which I run on yarn and sometimes it behaves in weird
manner it shows negative no of tasks in few executors and I keep on loosing
executors I also see no of executors are more than I requested. My job is
highly tuned not getting OOM or any problem. It is just YARN behaves in a
way sometimes so that executors keep on getting killed because of resource
crunching. Please guide how do I control YARN from behaving bad.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-0-running-jobs-in-yarn-shows-negative-no-of-tasks-in-executor-tp26337.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark 1.6.0 running jobs in yarn shows negative no of tasks in executor
Posted by Umesh Kacha <um...@gmail.com>.
Hi I am using Hadoop 2.4.0 it is not frequent sometimes it happens I dont
think my spark logic has any problem if logic would have been wrong it
would be failing everyday. I see mostly YARN killed executors so I see
executor lost in my driver logs.
On Thu, Feb 25, 2016 at 10:30 PM, Yin Yang <yy...@gmail.com> wrote:
> Which release of hadoop are you using ?
>
> Can you share a bit about the logic of your job ?
>
> Pastebinning portion of relevant logs would give us more clue.
>
> Thanks
>
> On Thu, Feb 25, 2016 at 8:54 AM, unk1102 <um...@gmail.com> wrote:
>
>> Hi I have spark job which I run on yarn and sometimes it behaves in weird
>> manner it shows negative no of tasks in few executors and I keep on
>> loosing
>> executors I also see no of executors are more than I requested. My job is
>> highly tuned not getting OOM or any problem. It is just YARN behaves in a
>> way sometimes so that executors keep on getting killed because of resource
>> crunching. Please guide how do I control YARN from behaving bad.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-0-running-jobs-in-yarn-shows-negative-no-of-tasks-in-executor-tp26337.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Spark 1.6.0 running jobs in yarn shows negative no of tasks in executor
Posted by Yin Yang <yy...@gmail.com>.
Which release of hadoop are you using ?
Can you share a bit about the logic of your job ?
Pastebinning portion of relevant logs would give us more clue.
Thanks
On Thu, Feb 25, 2016 at 8:54 AM, unk1102 <um...@gmail.com> wrote:
> Hi I have spark job which I run on yarn and sometimes it behaves in weird
> manner it shows negative no of tasks in few executors and I keep on loosing
> executors I also see no of executors are more than I requested. My job is
> highly tuned not getting OOM or any problem. It is just YARN behaves in a
> way sometimes so that executors keep on getting killed because of resource
> crunching. Please guide how do I control YARN from behaving bad.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-0-running-jobs-in-yarn-shows-negative-no-of-tasks-in-executor-tp26337.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>