[jira] [Commented] (MAHOUT-1802) Capture attached checkpoints (if cached)

["Hudson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234723#comment-15234723 ] 

Hudson commented on MAHOUT-1802:
--------------------------------

FAILURE: Integrated in Mahout-Quality #3324 (See [https://builds.apache.org/job/Mahout-Quality/3324/])
MAHOUT-1802: Capture attached checkpoints (if cached) closes (smarthi: rev 46071e6f3eb9ed53895222a54657326c83c62a53)
* h2o/src/main/scala/org/apache/mahout/h2obindings/drm/CheckpointedDrmH2O.scala
* math-scala/src/main/scala/org/apache/mahout/math/drm/DistributedEngine.scala
* h2o/src/main/scala/org/apache/mahout/h2obindings/H2OEngine.scala
* spark/src/main/scala/org/apache/mahout/sparkbindings/SparkEngine.scala
* spark/src/main/scala/org/apache/mahout/sparkbindings/drm/CheckpointedDrmSpark.scala
* math-scala/src/main/scala/org/apache/mahout/math/drm/CheckpointedDrm.scala
* spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala


>  Capture attached checkpoints (if cached)
> -----------------------------------------
>
>                 Key: MAHOUT-1802
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-1802
>             Project: Mahout
>          Issue Type: Improvement
>    Affects Versions: 0.11.1
>            Reporter: Andrew Palumbo
>            Assignee: Andrew Palumbo
>             Fix For: 0.11.2
>
>
> Currently, the optimizer generates checkpoints and attaches them to actual logical elements of the DAG via CheckpointAction$cp. 
> the way it worsk today is as follows: 
> {code}
> drmC = drmA+ drmB
> val cp1 = drmC.checkpoint() // checkpoint
> val cp2 = drmC.checkpoint() // cp2 == cp1
> drmD = cp1 + drmE // cp1 + drmE
> {code}
> but, in: 
> {code}
> drmD = drmC + drmE // computes drmA + drmB + drmC all over
> {code}
> {{drmC}} already has {{cp1}} attached to it so we should assume the common computational path is the intent here regardless and should be used, instead of building plans that recompute it. That is, 
> {{drmD = drmC + drmE}} should imply {{cp1 + drmE}} as well even if checkpoint is not used explicitly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)