You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by dachuan <hd...@gmail.com> on 2013/10/20 02:53:48 UTC

help me find the lineage graph code in spark

Hi, all,

I am pretty new to spark internal world, and currently I am interested in
the fault tolerance in spark.

a little surprised, i didn't find "lineage" keyword in directory
"incubator-spark/core/src/main/scala/org/apache/spark", I thought it must
be there because the nsdi 2012 paper mentioned it.

[huangda@dachuan-desktop spark]$ find -type f | xargs grep -in "lineage"
[huangda@dachuan-desktop spark]$

Could you please point me into the right source code file if you happen to
know?

thank you,
dachuan.


-- 
Dachuan Huang
Cellphone: 614-390-7234
2015 Neil Avenue
Ohio State University
Columbus, Ohio
U.S.A.
43210

Re: help me find the lineage graph code in spark

Posted by Dachuan Huang <hu...@cse.ohio-state.edu>.

Thanks!


On Sat, Oct 19, 2013 at 9:11 PM, Mark Hamstra <ma...@clearstorydata.com>wrote:

> What you are looking for is 'deps':
>
> abstract class RDD[T: ClassManifest](
>     @transient private var sc: SparkContext,
>     @transient private var deps: Seq[Dependency[_]]
>   ) extends Serializable with Logging {...}
>
>
>
> On Sat, Oct 19, 2013 at 5:53 PM, dachuan <hd...@gmail.com> wrote:
>
>> Hi, all,
>>
>> I am pretty new to spark internal world, and currently I am interested in
>> the fault tolerance in spark.
>>
>> a little surprised, i didn't find "lineage" keyword in directory
>> "incubator-spark/core/src/main/scala/org/apache/spark", I thought it must
>> be there because the nsdi 2012 paper mentioned it.
>>
>> [huangda@dachuan-desktop spark]$ find -type f | xargs grep -in "lineage"
>> [huangda@dachuan-desktop spark]$
>>
>> Could you please point me into the right source code file if you happen
>> to know?
>>
>> thank you,
>> dachuan.
>>
>>
>> --
>> Dachuan Huang
>> Cellphone: 614-390-7234
>> 2015 Neil Avenue
>> Ohio State University
>> Columbus, Ohio
>> U.S.A.
>> 43210
>>
>
>

Re: help me find the lineage graph code in spark

Posted by Mark Hamstra <ma...@clearstorydata.com>.

What you are looking for is 'deps':

abstract class RDD[T: ClassManifest](
    @transient private var sc: SparkContext,
    @transient private var deps: Seq[Dependency[_]]
  ) extends Serializable with Logging {...}



On Sat, Oct 19, 2013 at 5:53 PM, dachuan <hd...@gmail.com> wrote:

> Hi, all,
>
> I am pretty new to spark internal world, and currently I am interested in
> the fault tolerance in spark.
>
> a little surprised, i didn't find "lineage" keyword in directory
> "incubator-spark/core/src/main/scala/org/apache/spark", I thought it must
> be there because the nsdi 2012 paper mentioned it.
>
> [huangda@dachuan-desktop spark]$ find -type f | xargs grep -in "lineage"
> [huangda@dachuan-desktop spark]$
>
> Could you please point me into the right source code file if you happen to
> know?
>
> thank you,
> dachuan.
>
>
> --
> Dachuan Huang
> Cellphone: 614-390-7234
> 2015 Neil Avenue
> Ohio State University
> Columbus, Ohio
> U.S.A.
> 43210
>