[GitHub] spark pull request: [SPARK-12363][Mllib] Remove setRun and fix Pow...

[jkbradley <gi...@git.apache.org>]

Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/10539#issuecomment-169091413
  
    @ankurdave Would you mind checking out this issue?  It's really weird to me.  Here's what's happening:
    * Background: PowerIterationClustering (PIC) initializes a ```Graph[Double, Double]``` which represents a (symmetric) similarity matrix.  It then runs power iteration on the graph (i.e., on the matrix) to get an approximate first eigenvector.  It then clusters the vertices (based on their 1 Double value in the eigenvector) using KMeans.
    * On the surface: This PR changes KMeans in PIC to run 1 time instead of 5 times.  That changes the results of PIC.
    * Test failure: There are 2 comparable tests in PowerIterationClusteringSuite: ```test("power iteration clustering")``` and ```test("power iteration clustering on graph")```.  The second one is failing, even though it seems to begin with exactly the same graph and values.
    * If you look more closely, it seems to be an issue with partitioning:
      * Same initialization: One test gives PIC a list of similarities (which PIC converts to a graph), and the other gives PIC the graph.  However, if you print the graphs PIC begins with, they are exactly the same, except for the partitioning.
      * The PIC runs in the 2 tests diverge on the first iteration.
      * The 2 "fixes" I've seen are:
        * Changing ```TripletFields.Src``` to ```TripletFields.All``` somehow fixes the problem, even though only the edge and src attributes are used.  (done in this PR)
        * Changing the number of partitions from 2 to 1 also fixes the problem. (tested locally)
    
    Is it possible that GraphX is not shipping the correct data between partitions?  Thanks a lot in advance for your help!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org