You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by on 2015/08/06 23:19:34 UTC
OOM with Zeppelin Spark Query
Data Size: 167 MB
Schema:
case class GuidSess(
guid: String,
sessionKey: String,
sessionStartDate: String,
siteId: String,
eventCount: String,
browser: String,
browserVersion: String,
operatingSystem: String,
experimentChannel: String,
deviceName: String)
%sql
select siteid, count(distinct guid) total_visitor,
count(sessionKey) as total_visits
from guidsess
group by siteid
This started a job with 1 stage and 1 task and 1 executor
The task failed twice and i clicked || button on zeppelin and cancelled the
job. In logs i see
java.lang.OutOfMemoryError: GC overhead limit exceeded
Question
1) How can i increase the number of executors ? I thought this was dynamic
and handled by Zeppelin itself.
2) How can i increase the memory for each executor ?
Parallel Question
1) A analytics user (the one who fires all SQL like queries) need/will not
have knowledge about underlying processing framework
(Spark/Teradata/Hadoop/XYZ) . Once data is loaded (the code written by a
Spark/Scala developer) the analytic person should be able to run any kind
of SQL queries to analyze the data.
2) Should not Zeppelin handle the Spark optimization/tuning parameters ?
3) The queries are dynamic and will change. If each query requires tuning
spark then it might be a involved activity.
Please suggest.
-Deepak
--
Deepak
Re: OOM with Zeppelin Spark Query
Posted by .
There is 1 Job, 2 stages with 1 task each.
*1st Stage : mapPartitions*
[image: Inline image 1]
1st Stage: Task 1 started to fail. A second attempt started for 1st task of
first Stage. The first attempt failed "Executor LOST"
when i go to YARN resource manager and go to that particular host, i see
that its running fine.
*Attempt #1*
[image: Inline image 2]
*Attempt #2* Executor LOST AGAIN
[image: Inline image 3]
*Attempt 3&4*
*[image: Inline image 4]*
*2nd Stage runJob : SKIPPED*
*[image: Inline image 5]*
Any suggestions ?
On Thu, Aug 6, 2015 at 2:19 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Data Size: 167 MB
>
> Schema:
>
> case class GuidSess(
>
> guid: String,
>
> sessionKey: String,
>
> sessionStartDate: String,
>
> siteId: String,
>
> eventCount: String,
>
> browser: String,
>
> browserVersion: String,
>
> operatingSystem: String,
>
> experimentChannel: String,
>
> deviceName: String)
>
>
> %sql
> select siteid, count(distinct guid) total_visitor,
> count(sessionKey) as total_visits
> from guidsess
> group by siteid
>
>
> This started a job with 1 stage and 1 task and 1 executor
>
> The task failed twice and i clicked || button on zeppelin and cancelled
> the job. In logs i see
>
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Question
> 1) How can i increase the number of executors ? I thought this was dynamic
> and handled by Zeppelin itself.
> 2) How can i increase the memory for each executor ?
>
> Parallel Question
> 1) A analytics user (the one who fires all SQL like queries) need/will not
> have knowledge about underlying processing framework
> (Spark/Teradata/Hadoop/XYZ) . Once data is loaded (the code written by a
> Spark/Scala developer) the analytic person should be able to run any kind
> of SQL queries to analyze the data.
> 2) Should not Zeppelin handle the Spark optimization/tuning parameters ?
> 3) The queries are dynamic and will change. If each query requires tuning
> spark then it might be a involved activity.
>
> Please suggest.
>
> -Deepak
>
> --
> Deepak
>
>
--
Deepak