You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sandip Mehta <sa...@gmail.com> on 2015/10/02 00:06:18 UTC
Call Site - Spark Context
Hi,
I wanted to understand what is the purpose of Call Site in Spark Context?
Regards
SM
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Call Site - Spark Context
Posted by Sandip Mehta <sa...@gmail.com>.
Thanks Robin.
Regards
SM
> On 01-Oct-2015, at 3:15 pm, Robin East <ro...@xense.co.uk> wrote:
>
> From the comments in the code:
>
> When called inside a class in the spark package, returns the name of the user code class (outside the spark package) that called into Spark, as well as which Spark method they called. This is used, for example, to tell users where in their code each RDD got created.
> Keep crawling up the stack trace until we find the first function not inside of the spark package. We track the last (shallowest) contiguous Spark method. This might be an RDD transformation, a SparkContext function (such as parallelize), or anything else that leads to instantiation of an RDD. We also track the first (deepest) user method, file, and line.
> So basically it’s a mechanism to report where in the user’s code an RDD is created.
> -------------------------------------------------------------------------------
> Robin East
> Spark GraphX in Action Michael Malak and Robin East
> Manning Publications Co.
> http://www.manning.com/books/spark-graphx-in-action <http://www.manning.com/books/spark-graphx-in-action>
>
>
>
>
>
>> On 1 Oct 2015, at 23:06, Sandip Mehta <sandip.mehta.sub@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi,
>>
>> I wanted to understand what is the purpose of Call Site in Spark Context?
>>
>> Regards
>> SM
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
>>
>
Re: Call Site - Spark Context
Posted by Sandip Mehta <sa...@gmail.com>.
Thanks Robin.
Regards
SM
> On 01-Oct-2015, at 3:15 pm, Robin East <ro...@xense.co.uk> wrote:
>
> From the comments in the code:
>
> When called inside a class in the spark package, returns the name of the user code class (outside the spark package) that called into Spark, as well as which Spark method they called. This is used, for example, to tell users where in their code each RDD got created.
> Keep crawling up the stack trace until we find the first function not inside of the spark package. We track the last (shallowest) contiguous Spark method. This might be an RDD transformation, a SparkContext function (such as parallelize), or anything else that leads to instantiation of an RDD. We also track the first (deepest) user method, file, and line.
> So basically it’s a mechanism to report where in the user’s code an RDD is created.
> -------------------------------------------------------------------------------
> Robin East
> Spark GraphX in Action Michael Malak and Robin East
> Manning Publications Co.
> http://www.manning.com/books/spark-graphx-in-action <http://www.manning.com/books/spark-graphx-in-action>
>
>
>
>
>
>> On 1 Oct 2015, at 23:06, Sandip Mehta <sandip.mehta.sub@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi,
>>
>> I wanted to understand what is the purpose of Call Site in Spark Context?
>>
>> Regards
>> SM
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
>>
>