You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by on 2015/08/06 23:19:34 UTC

OOM with Zeppelin Spark Query

Data Size: 167 MB

Schema:

 case class GuidSess(

      guid: String,

      sessionKey: String,

      sessionStartDate: String,

      siteId: String,

      eventCount: String,

      browser: String,

      browserVersion: String,

      operatingSystem: String,

      experimentChannel: String,

      deviceName: String)


%sql
select siteid, count(distinct guid) total_visitor,
count(sessionKey) as total_visits
from guidsess
group by siteid


This started a job with 1 stage and 1 task and 1 executor

The task failed twice and i clicked || button on zeppelin and cancelled the
job. In logs i see

java.lang.OutOfMemoryError: GC overhead limit exceeded

Question
1) How can i increase the number of executors ? I thought this was dynamic
and handled by Zeppelin itself.
2) How can i increase the memory for each executor ?

Parallel Question
1) A analytics user (the one who fires all SQL like queries) need/will not
have knowledge about underlying processing framework
(Spark/Teradata/Hadoop/XYZ) . Once data is loaded (the code written by a
Spark/Scala developer) the analytic person should be able to run any kind
of SQL queries to analyze the data.
2) Should not Zeppelin handle the Spark optimization/tuning parameters ?
3) The queries are dynamic and will change. If each query requires tuning
spark then it might be a involved activity.

Please suggest.

-Deepak

-- 
Deepak

Re: OOM with Zeppelin Spark Query

Posted by .
There is 1 Job, 2 stages with 1 task each.

*1st Stage : mapPartitions*
[image: Inline image 1]

1st Stage: Task 1 started to fail. A second attempt started for 1st task of
first Stage. The first attempt failed "Executor LOST"
when i go to YARN resource manager and go to that particular host, i see
that its running fine.

*Attempt #1*
[image: Inline image 2]

*Attempt #2* Executor LOST AGAIN
[image: Inline image 3]
*Attempt 3&4*

*[image: Inline image 4]*



*2nd Stage runJob : SKIPPED*

*[image: Inline image 5]*

Any suggestions ?




On Thu, Aug 6, 2015 at 2:19 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:

> Data Size: 167 MB
>
> Schema:
>
>  case class GuidSess(
>
>       guid: String,
>
>       sessionKey: String,
>
>       sessionStartDate: String,
>
>       siteId: String,
>
>       eventCount: String,
>
>       browser: String,
>
>       browserVersion: String,
>
>       operatingSystem: String,
>
>       experimentChannel: String,
>
>       deviceName: String)
>
>
> %sql
> select siteid, count(distinct guid) total_visitor,
> count(sessionKey) as total_visits
> from guidsess
> group by siteid
>
>
> This started a job with 1 stage and 1 task and 1 executor
>
> The task failed twice and i clicked || button on zeppelin and cancelled
> the job. In logs i see
>
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Question
> 1) How can i increase the number of executors ? I thought this was dynamic
> and handled by Zeppelin itself.
> 2) How can i increase the memory for each executor ?
>
> Parallel Question
> 1) A analytics user (the one who fires all SQL like queries) need/will not
> have knowledge about underlying processing framework
> (Spark/Teradata/Hadoop/XYZ) . Once data is loaded (the code written by a
> Spark/Scala developer) the analytic person should be able to run any kind
> of SQL queries to analyze the data.
> 2) Should not Zeppelin handle the Spark optimization/tuning parameters ?
> 3) The queries are dynamic and will change. If each query requires tuning
> spark then it might be a involved activity.
>
> Please suggest.
>
> -Deepak
>
> --
> Deepak
>
>


-- 
Deepak