Is there a way to look at RDD's lineage? Or debug a fault-tolerance error?

[Sung Hwan Chung <co...@cs.stanford.edu>]

My job is not being fault-tolerant (e.g., when there's a fetch failure or
something).

The lineage of RDDs are constantly updated every iteration. However, I
think that when there's a failure, the lineage information is not being
correctly reapplied.

It goes something like this:

val rawRDD = read(...)
val repartRDD = rawRDD.repartition(X)

val tx1 = repartRDD.map(...)
var tx2 = tx1.map(...)

while (...) {
  tx2 = tx1.zip(tx2).map(...)
}


Is there any way to monitor RDD's lineage, maybe even including? I want to
make sure that there's no unexpected things happening.

Re: Is there a way to look at RDD's lineage? Or debug a fault-tolerance error?

[Pierre B <pi...@realimpactanalytics.com>]

To add a bit on this one, if you look at RDD.scala in Spark code, you'll see
that both "parent" and "firstParent" methods are protected[spark]. 

I guess, for good reasons, that I must admit I don't understand completely,
you are not supposed to explore an RDD lineage programmatically...

I had a usecase myself and was disappointed to find out about this.

Anyone could enlighten me on the reasons for this restriction?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-way-to-look-at-RDD-s-lineage-Or-debug-a-fault-tolerance-error-tp15959p16041.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Is there a way to look at RDD's lineage? Or debug a fault-tolerance error?

[Sung Hwan Chung <co...@cs.stanford.edu>]

One thing I didn't mention is that we actually do data.repartition before
hand with shuffle.

I found that this can actually introduce randomness to lineage steps,
because data get shuffled to different partitions and lead to inconsistent
behavior if your algorithm is dependent on the order at which the data rows
appear, because now data rows will appear in a different orders.

If you want to guarantee fault-tolerance, you can't have any randomness
whatsoever in lineage steps, and repartition violates that (depending on
what you do with the data).

On Wed, Oct 8, 2014 at 12:24 PM, Sung Hwan Chung <co...@gmail.com> wrote:

> There is no circular dependency. Its simply dropping references to prev
> RDDs because there is no need for it.
>
> I wonder if that messes up things up though internally for Spark due to
> losing references to intermediate RDDs.
>
> On Oct 8, 2014, at 12:13 PM, Akshat Aranya <aa...@gmail.com> wrote:
>
> Using a var for RDDs in this way is not going to work.  In this example,
> tx1.zip(tx2) would create and RDD that depends on tx2, but then soon after
> that, you change what tx2 means, so you would end up having a circular
> dependency.
>
> On Wed, Oct 8, 2014 at 12:01 PM, Sung Hwan Chung <codedeft@cs.stanford.edu
> > wrote:
>
>> My job is not being fault-tolerant (e.g., when there's a fetch failure or
>> something).
>>
>> The lineage of RDDs are constantly updated every iteration. However, I
>> think that when there's a failure, the lineage information is not being
>> correctly reapplied.
>>
>> It goes something like this:
>>
>> val rawRDD = read(...)
>> val repartRDD = rawRDD.repartition(X)
>>
>> val tx1 = repartRDD.map(...)
>> var tx2 = tx1.map(...)
>>
>> while (...) {
>>   tx2 = tx1.zip(tx2).map(...)
>> }
>>
>>
>> Is there any way to monitor RDD's lineage, maybe even including? I want
>> to make sure that there's no unexpected things happening.
>>
>
>

Re: Is there a way to look at RDD's lineage? Or debug a fault-tolerance error?

[Sung Hwan Chung <co...@gmail.com>]

There is no circular dependency. Its simply dropping references to prev RDDs because there is no need for it.

I wonder if that messes up things up though internally for Spark due to losing references to intermediate RDDs.

> On Oct 8, 2014, at 12:13 PM, Akshat Aranya <aa...@gmail.com> wrote:
> 
> Using a var for RDDs in this way is not going to work.  In this example, tx1.zip(tx2) would create and RDD that depends on tx2, but then soon after that, you change what tx2 means, so you would end up having a circular dependency.
> 
>> On Wed, Oct 8, 2014 at 12:01 PM, Sung Hwan Chung <co...@cs.stanford.edu> wrote:
>> My job is not being fault-tolerant (e.g., when there's a fetch failure or something).
>> 
>> The lineage of RDDs are constantly updated every iteration. However, I think that when there's a failure, the lineage information is not being correctly reapplied.
>> 
>> It goes something like this:
>> 
>> val rawRDD = read(...)
>> val repartRDD = rawRDD.repartition(X)
>> 
>> val tx1 = repartRDD.map(...)
>> var tx2 = tx1.map(...)
>> 
>> while (...) {
>>   tx2 = tx1.zip(tx2).map(...)
>> }
>> 
>> 
>> Is there any way to monitor RDD's lineage, maybe even including? I want to make sure that there's no unexpected things happening.
> 

Re: Is there a way to look at RDD's lineage? Or debug a fault-tolerance error?

[Akshat Aranya <aa...@gmail.com>]

Using a var for RDDs in this way is not going to work.  In this example,
tx1.zip(tx2) would create and RDD that depends on tx2, but then soon after
that, you change what tx2 means, so you would end up having a circular
dependency.

On Wed, Oct 8, 2014 at 12:01 PM, Sung Hwan Chung <co...@cs.stanford.edu>
wrote:

> My job is not being fault-tolerant (e.g., when there's a fetch failure or
> something).
>
> The lineage of RDDs are constantly updated every iteration. However, I
> think that when there's a failure, the lineage information is not being
> correctly reapplied.
>
> It goes something like this:
>
> val rawRDD = read(...)
> val repartRDD = rawRDD.repartition(X)
>
> val tx1 = repartRDD.map(...)
> var tx2 = tx1.map(...)
>
> while (...) {
>   tx2 = tx1.zip(tx2).map(...)
> }
>
>
> Is there any way to monitor RDD's lineage, maybe even including? I want to
> make sure that there's no unexpected things happening.
>

Re: Is there a way to look at RDD's lineage? Or debug a fault-tolerance error?

[Liquan Pei <li...@gmail.com>]

There is a toDebugString method in rdd that will print a description of
this RDD and its recursive dependencies for debugging.

Thanks,
Liquan

On Wed, Oct 8, 2014 at 12:01 PM, Sung Hwan Chung <co...@cs.stanford.edu>
wrote:

> My job is not being fault-tolerant (e.g., when there's a fetch failure or
> something).
>
> The lineage of RDDs are constantly updated every iteration. However, I
> think that when there's a failure, the lineage information is not being
> correctly reapplied.
>
> It goes something like this:
>
> val rawRDD = read(...)
> val repartRDD = rawRDD.repartition(X)
>
> val tx1 = repartRDD.map(...)
> var tx2 = tx1.map(...)
>
> while (...) {
>   tx2 = tx1.zip(tx2).map(...)
> }
>
>
> Is there any way to monitor RDD's lineage, maybe even including? I want to
> make sure that there's no unexpected things happening.
>



-- 
Liquan Pei
Department of Physics
University of Massachusetts Amherst