[jira] [Closed] (SPARK-6697) PeriodicGraphCheckpointer is not clear edges.

["Joseph K. Bradley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joseph K. Bradley closed SPARK-6697.
------------------------------------
    Resolution: Not A Problem

I'm going to close this for now since it's not really a bug, just a missing feature in a private API.  If this is needed for something, feel free to create a new JIRA for adding the feature.

> PeriodicGraphCheckpointer is not clear edges.
> ---------------------------------------------
>
>                 Key: SPARK-6697
>                 URL: https://issues.apache.org/jira/browse/SPARK-6697
>             Project: Spark
>          Issue Type: Bug
>          Components: GraphX, MLlib
>    Affects Versions: 1.3.0
>            Reporter: Guoqiang Li
>         Attachments: QQ20150403-1.png
>
>
> When I run this [branch(lrGraphxSGD)| https://github.com/witgo/spark/tree/lrGraphxSGD] .
> PeriodicGraphCheckpointer only clear the vertices.
> {code} 
> def run(iterations: Int): Unit = {
>     for (iter <- 1 to iterations) {
>       logInfo(s"Start train (Iteration $iter/$iterations)")
>       val margin = forward()
>       margin.setName(s"margin-$iter").persist(storageLevel)
>       println(s"train (Iteration $iter/$iterations) cost : ${error(margin)}")
>       var gradient = backward(margin)
>       gradient = updateDeltaSum(gradient, iter)
>       dataSet = updateWeight(gradient, iter)
>       dataSet.vertices.setName(s"vertices-$iter")
>       dataSet.edges.setName(s"edges-$iter")
>       dataSet.persist(storageLevel)
>       graphCheckpointer.updateGraph(dataSet)
>       margin.unpersist(blocking = false)
>       gradient.unpersist(blocking = false)
>       logInfo(s"End train (Iteration $iter/$iterations)")
>       innerIter += 1
>     }
>     graphCheckpointer.deleteAllCheckpoints()
>   }
>   // Updater for L1 regularized problems
>   private def updateWeight(delta: VertexRDD[Double], iter: Int): Graph[VD, ED] = {
>     val thisIterStepSize = if (useAdaGrad) stepSize else stepSize / sqrt(iter)
>     val thisIterL1StepSize = stepSize / sqrt(iter)
>     val newVertices = dataSet.vertices.leftJoin(delta) { (_, attr, gradient) =>
>       gradient match {
>         case Some(gard) => {
>           var weight = attr
>           weight -= thisIterStepSize * gard
>           if (regParam > 0.0 && weight != 0.0) {
>             val shrinkageVal = regParam * thisIterL1StepSize
>             weight = signum(weight) * max(0.0, abs(weight) - shrinkageVal)
>           }
>           assert(!weight.isNaN)
>           weight
>         }
>         case None => attr
>       }
>     }
>     GraphImpl(newVertices, dataSet.edges)
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org