Input parsing time

[Carlos Eduardo Santos <ce...@gmail.com>]

Hi,

I only loading a JSON and running one query. I would like to know how much
time is spent on reading, decompressing (e.g. bz2 file) and parsing the
file before the query begins to execute. I have the impression that all
processing time (parsing the input and running the query) is included in
the "Executor Computing Time" in History Server.

Do you recommend any documentation to understand better the History Server
logs and maybe more stats included in the log files?

Thanks in advance,
Carlos Eduardo M. Santos
CS PhD student

Re: Input parsing time

[Adrian Tanase <at...@adobe.com>]

You’re right – everything is captured under Executor Computing Time if it’s your app code.

I know that some people have used custom builds of spark that add more timers – they will show-up nicely in the Spark UI.

A more light-weight approach is to time it yourself via some counters / accumulators and log that information.
http://spark.apache.org/docs/latest/programming-guide.html#accumulators-a-nameaccumlinka

-adrian

From: Carlos Eduardo Santos
Reply-To: "cems@cemshost.com.br<ma...@cemshost.com.br>"
Date: Thursday, September 17, 2015 at 10:09 AM
To: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Input parsing time

Hi,

I only loading a JSON and running one query. I would like to know how much time is spent on reading, decompressing (e.g. bz2 file) and parsing the file before the query begins to execute. I have the impression that all processing time (parsing the input and running the query) is included in the "Executor Computing Time" in History Server.

Do you recommend any documentation to understand better the History Server logs and maybe more stats included in the log files?

Thanks in advance,
Carlos Eduardo M. Santos
CS PhD student