You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Debasish Das <de...@gmail.com> on 2014/08/15 21:30:57 UTC

ALS checkpoint performance

Hi,

Are there any experiments detailing the performance hit due to HDFS
checkpoint in ALS ?

As we scale to large ranks with more ratings, I believe we have to cut the
RDD lineage to safe guard against the lineage issue...

Thanks.
Deb

Re: ALS checkpoint performance

Posted by Xiangrui Meng <me...@gmail.com>.
Guoqiang reported some results in his PRs
https://github.com/apache/spark/pull/828 and
https://github.com/apache/spark/pull/929 . But this is really
problem-dependent. -Xiangrui

On Fri, Aug 15, 2014 at 12:30 PM, Debasish Das <de...@gmail.com> wrote:
> Hi,
>
> Are there any experiments detailing the performance hit due to HDFS
> checkpoint in ALS ?
>
> As we scale to large ranks with more ratings, I believe we have to cut the
> RDD lineage to safe guard against the lineage issue...
>
> Thanks.
> Deb
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org