You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by dash <bs...@nd.edu> on 2014/06/23 00:14:46 UTC
Checkpointed RDD still causing StackOverflow
Hi,
I'm doing iterative computing now, and due to lineage chain, we need to
checkpoint the RDD in order to cut off lineage and prevent StackOverflow
error.
The following code still having StackOverflowError, I checked
`isCheckpointed` and the result is true. Also, I write a function to count
the lineage, but the lineage is not big. Any idea about that? Please give me
some hit so I can dig into the source code and try to fix it.
Best,
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Checkpointed-RDD-still-causing-StackOverflow-tp7066.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: Checkpointed RDD still causing StackOverflow
Posted by dash <bs...@nd.edu>.
Due to SPARK-2245, you can not use count to materialize VertexRDD. That
actually materialize PartitionRDD, so checkpoint for VertexRDD won't work.
I'll trying to fix that right now.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Checkpointed-RDD-still-causing-StackOverflow-tp7066p7079.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: Checkpointed RDD still causing StackOverflow
Posted by Mayur Rustagi <ma...@gmail.com>.
Do not call collect as that will perform materialization as well as
transfer of data to driver (might actually cause driver to fail if the data
is huge). You have to materialize the RDD in some way(call save, count,
collect).
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Tue, Jun 24, 2014 at 2:50 AM, Xiangrui Meng <me...@gmail.com> wrote:
> Calling checkpoint() alone doesn't cut the lineage. It only marks the
> RDD as to be checkpointed. The lineage is cut after the first time
> this RDD is materialized. You see StackOverflow becaure the lineage is
> still there. -Xiangrui
>
> On Sun, Jun 22, 2014 at 6:37 PM, dash <bs...@nd.edu> wrote:
> > Hi Xiangrui,
> >
> > According to my knowledge, calling count is for materialize the RDD, does
> > collect do the same thing since it also an action? I can not call count
> > because for a Graph object, count does not materialize the RDD. I already
> > send an issue on that.
> >
> > My question is, why there still have stack overflow even if
> `isCheckpointed`
> > is true?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Checkpointed-RDD-still-causing-StackOverflow-tp7066p7068.html
> > Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
Re: Checkpointed RDD still causing StackOverflow
Posted by Xiangrui Meng <me...@gmail.com>.
Calling checkpoint() alone doesn't cut the lineage. It only marks the
RDD as to be checkpointed. The lineage is cut after the first time
this RDD is materialized. You see StackOverflow becaure the lineage is
still there. -Xiangrui
On Sun, Jun 22, 2014 at 6:37 PM, dash <bs...@nd.edu> wrote:
> Hi Xiangrui,
>
> According to my knowledge, calling count is for materialize the RDD, does
> collect do the same thing since it also an action? I can not call count
> because for a Graph object, count does not materialize the RDD. I already
> send an issue on that.
>
> My question is, why there still have stack overflow even if `isCheckpointed`
> is true?
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Checkpointed-RDD-still-causing-StackOverflow-tp7066p7068.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: Checkpointed RDD still causing StackOverflow
Posted by dash <bs...@nd.edu>.
Hi Xiangrui,
According to my knowledge, calling count is for materialize the RDD, does
collect do the same thing since it also an action? I can not call count
because for a Graph object, count does not materialize the RDD. I already
send an issue on that.
My question is, why there still have stack overflow even if `isCheckpointed`
is true?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Checkpointed-RDD-still-causing-StackOverflow-tp7066p7068.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: Checkpointed RDD still causing StackOverflow
Posted by Xiangrui Meng <me...@gmail.com>.
After checkpoint(), please call count(). This is similar to cache(),
the RDD is only marked as to be checked with checkpoint(). -Xiangrui
On Sun, Jun 22, 2014 at 3:14 PM, dash <bs...@nd.edu> wrote:
> Hi,
>
> I'm doing iterative computing now, and due to lineage chain, we need to
> checkpoint the RDD in order to cut off lineage and prevent StackOverflow
> error.
>
> The following code still having StackOverflowError, I checked
> `isCheckpointed` and the result is true. Also, I write a function to count
> the lineage, but the lineage is not big. Any idea about that? Please give me
> some hit so I can dig into the source code and try to fix it.
>
>
>
>
> Best,
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Checkpointed-RDD-still-causing-StackOverflow-tp7066.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.