You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by huaxingao <gi...@git.apache.org> on 2018/06/08 19:54:25 UTC

[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

GitHub user huaxingao opened a pull request:

    https://github.com/apache/spark/pull/21513

    [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

    ## What changes were proposed in this pull request?
    
    add spark.ml Python API for PIC
    
    ## How was this patch tested?
    
    add doctest


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/huaxingao/spark spark--19826

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21513
    
----
commit cdc97c103ea3f3da60e5e93482a7994f94e790c8
Author: Huaxin Gao <hu...@...>
Date:   2018-06-08T19:51:43Z

    [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194244595
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,213 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use :py:func:`assignClusters` method
    +    to run the PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +   >>> data = [((long)(1), (long)(0), 0.5), \
    +               ((long)(2), (long)(0), 0.5), \
    +               ((long)(2), (long)(1), 0.7), \
    +               ((long)(3), (long)(0), 0.5), \
    +               ((long)(3), (long)(1), 0.7), \
    +               ((long)(3), (long)(2), 0.9), \
    +               ((long)(4), (long)(0), 0.5), \
    +               ((long)(4), (long)(1), 0.7), \
    +               ((long)(4), (long)(2), 0.9), \
    +               ((long)(4), (long)(3), 1.1), \
    +               ((long)(5), (long)(0), 0.5), \
    +               ((long)(5), (long)(1), 0.7), \
    +               ((long)(5), (long)(2), 0.9), \
    +               ((long)(5), (long)(3), 1.1), \
    +               ((long)(5), (long)(4), 1.3)]
    +    >>> df = spark.createDataFrame(data).toDF("src", "dst", "weight")
    +    >>> pic = PowerIterationClustering()
    --- End diff --
    
    If we only keep one example, we should use keyword args:
    
    ~~~python
    pic = PowerIterationClustering(k=2, maxIter=40, weightCol="weight")
    assignments = pic.assignClusters(df)
    ~~~


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3866/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Thanks a lot for your help! @mengxr @WeichenXu123 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194244698
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,213 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use :py:func:`assignClusters` method
    +    to run the PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +   >>> data = [((long)(1), (long)(0), 0.5), \
    +               ((long)(2), (long)(0), 0.5), \
    +               ((long)(2), (long)(1), 0.7), \
    +               ((long)(3), (long)(0), 0.5), \
    +               ((long)(3), (long)(1), 0.7), \
    +               ((long)(3), (long)(2), 0.9), \
    +               ((long)(4), (long)(0), 0.5), \
    +               ((long)(4), (long)(1), 0.7), \
    +               ((long)(4), (long)(2), 0.9), \
    +               ((long)(4), (long)(3), 1.1), \
    +               ((long)(5), (long)(0), 0.5), \
    +               ((long)(5), (long)(1), 0.7), \
    +               ((long)(5), (long)(2), 0.9), \
    +               ((long)(5), (long)(3), 1.1), \
    +               ((long)(5), (long)(4), 1.3)]
    +    >>> df = spark.createDataFrame(data).toDF("src", "dst", "weight")
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> assignments.sort(assignments.id).show(truncate=False)
    +    +---+-------+
    +    |id |cluster|
    +    +---+-------+
    +    |0  |1      |
    +    |1  |1      |
    +    |2  |1      |
    +    |3  |1      |
    +    |4  |1      |
    +    |5  |0      |
    +    +---+-------+
    +    ...
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    --- End diff --
    
    This and `pic3` seem unnecessary to me as doctest. Same for pic3. Doctest is mainly to provide examples, not a full suite of unit tests.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194190113
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1159,216 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    +    PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +    >>> from pyspark.sql.types import StructField, StructType
    +    >>> import math
    +    >>> def genCircle(r, n):
    +    ...     points = []
    +    ...     for i in range(0, n):
    +    ...         theta = 2.0 * math.pi * i / n
    +    ...         points.append((r * math.cos(theta), r * math.sin(theta)))
    +    ...     return points
    +    >>> def sim(x, y):
    +    ...     dist = (x[0] - y[0]) * (x[0] - y[0]) + (x[1] - y[1]) * (x[1] - y[1])
    +    ...     return math.exp(-dist / 2.0)
    +    >>> r1 = 1.0
    +    >>> n1 = 10
    +    >>> r2 = 4.0
    +    >>> n2 = 40
    +    >>> n = n1 + n2
    +    >>> points = genCircle(r1, n1) + genCircle(r2, n2)
    +    >>> data = [(i, j, sim(points[i], points[j])) for i in range(1, n) for j in range(0, i)]
    +    >>> rdd = sc.parallelize(data, 2)
    +    >>> schema = StructType([StructField("src", LongType(), False), \
    +                 StructField("dst", LongType(),  True), \
    +                 StructField("weight", DoubleType(), True)])
    +    >>> df = spark.createDataFrame(rdd, schema)
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> result = sorted(assignments.collect(), key=lambda x: x.id)
    +    >>> result[0].cluster == result[1].cluster == result[2].cluster == result[3].cluster
    +    True
    +    >>> result[4].cluster == result[5].cluster == result[6].cluster == result[7].cluster
    +    True
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    +    >>> result2 = sorted(assignments2.collect(), key=lambda x: x.id)
    +    >>> result2[0].cluster == result2[1].cluster == result2[2].cluster == result2[3].cluster
    +    True
    +    >>> result2[4].cluster == result2[5].cluster == result2[6].cluster == result2[7].cluster
    +    True
    +    >>> pic3 = PowerIterationClustering(k=4, initMode="degree", srcCol="source", dstCol="dest")
    +    >>> pic3.getSrcCol()
    +    'source'
    +    >>> pic3.getDstCol()
    +    'dest'
    +    >>> pic3.getK()
    +    4
    +    >>> pic3.getMaxIter()
    +    20
    +    >>> pic3.getInitMode()
    +    'degree'
    +
    +    .. versionadded:: 2.4.0
    +    """
    +
    +    k = Param(Params._dummy(), "k",
    +              "The number of clusters to create. Must be > 1.",
    +              typeConverter=TypeConverters.toInt)
    +    initMode = Param(Params._dummy(), "initMode",
    +                     "The initialization algorithm. This can be either " +
    +                     "'random' to use a random vector as vertex properties, or 'degree' to use " +
    +                     "a normalized sum of similarities with other vertices.  Supported options: " +
    +                     "'random' and 'degree'.",
    +                     typeConverter=TypeConverters.toString)
    +    srcCol = Param(Params._dummy(), "srcCol",
    +                   "Name of the input column for source vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +    dstCol = Param(Params._dummy(), "dstCol",
    +                   "Name of the input column for destination vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +
    +    @keyword_only
    +    def __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                 weightCol=None):
    +        """
    +        __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                 weightCol=None)
    +        """
    +        super(PowerIterationClustering, self).__init__()
    +        self._java_obj = self._new_java_obj(
    +            "org.apache.spark.ml.clustering.PowerIterationClustering", self.uid)
    +        self._setDefault(k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst")
    +        kwargs = self._input_kwargs
    +        self.setParams(**kwargs)
    +
    +    @keyword_only
    +    @since("2.4.0")
    +    def setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                  weightCol=None):
    +        """
    +        setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                  weightCol=None)
    +        Sets params for PowerIterationClustering.
    +        """
    +        kwargs = self._input_kwargs
    +        return self._set(**kwargs)
    +
    +    @since("2.4.0")
    +    def setK(self, value):
    +        """
    +        Sets the value of :py:attr:`k`.
    +        """
    +        return self._set(k=value)
    +
    +    @since("2.4.0")
    +    def getK(self):
    +        """
    +        Gets the value of :py:attr:`k`.
    +        """
    +        return self.getOrDefault(self.k)
    +
    +    @since("2.4.0")
    +    def setInitMode(self, value):
    +        """
    +        Sets the value of :py:attr:`initMode`.
    +        """
    +        return self._set(initMode=value)
    +
    +    @since("2.4.0")
    +    def getInitMode(self):
    +        """
    +        Gets the value of `initMode`
    +        """
    +        return self.getOrDefault(self.initMode)
    +
    +    @since("2.4.0")
    +    def setSrcCol(self, value):
    +        """
    +        Sets the value of :py:attr:`srcCol`.
    +        """
    +        return self._set(srcCol=value)
    +
    +    @since("2.4.0")
    +    def getSrcCol(self):
    +        """
    +        Gets the value of :py:attr:`srcCol`.
    +        """
    +        return self.getOrDefault(self.srcCol)
    +
    +    @since("2.4.0")
    +    def setDstCol(self, value):
    +        """
    +        Sets the value of :py:attr:`dstCol`.
    +        """
    +        return self._set(dstCol=value)
    +
    +    @since("2.4.0")
    +    def getDstCol(self):
    +        """
    +        Gets the value of :py:attr:`dstCol`.
    +        """
    +        return self.getOrDefault(self.dstCol)
    +
    +    @since("2.4.0")
    +    def assignClusters(self, dataset):
    +        """
    +        Run the PIC algorithm and returns a cluster assignment for each input vertex.
    +
    +        :param dataset:
    +          A dataset with columns src, dst, weight representing the affinity matrix,
    +          which is the matrix A in the PIC paper. Suppose the src column value is i,
    +          the dst column value is j, the weight column value is similarity s,,ij,,
    +          which must be nonnegative. This is a symmetric matrix and hence
    +          s,,ij,, = s,,ji,,. For any (i, j) with nonzero similarity, there should be
    +          either (i, j, s,,ij,,) or (j, i, s,,ji,,) in the input. Rows with i = j are
    +          ignored, because we assume s,,ij,, = 0.0.
    +
    +        :return: A dataset that contains columns of vertex id and the corresponding cluster for
    +                 the id. The schema of it will be:
    +                   - id: Long
    +                   - cluster: Int
    +        """
    +        weightCol = None
    +        w = None
    +        if (self.isDefined(self.weightCol)):
    +            weightCol = self.getWeightCol()
    +        if (weightCol is None or len(weightCol) == 0):
    +            w = lit(1.0)
    +        else:
    +            w = col(weightCol).cast(DoubleType())
    +        srcCol = self.getSrcCol()
    +        dstCol = self.getDstCol()
    +        df = dataset.select(col(srcCol).cast(LongType()), col(dstCol).cast(LongType()), w)
    +        data = df.rdd.map(lambda x: (x[0], x[1], x[2]))
    +        algorithm = MLlibPowerIterationClustering()
    --- End diff --
    
    @WeichenXu123 Thank you very much for your comment. Will change now. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194237396
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,204 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    +    PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +    >>> from pyspark.sql.types import DoubleType, LongType, StructField, StructType
    +    >>> import math
    +    >>> def genCircle(r, n):
    +    ...     points = []
    +    ...     for i in range(0, n):
    +    ...         theta = 2.0 * math.pi * i / n
    +    ...         points.append((r * math.cos(theta), r * math.sin(theta)))
    +    ...     return points
    +    >>> def sim(x, y):
    +    ...     dist = (x[0] - y[0]) * (x[0] - y[0]) + (x[1] - y[1]) * (x[1] - y[1])
    +    ...     return math.exp(-dist / 2.0)
    +    >>> r1 = 1.0
    +    >>> n1 = 10
    +    >>> r2 = 4.0
    +    >>> n2 = 40
    +    >>> n = n1 + n2
    +    >>> points = genCircle(r1, n1) + genCircle(r2, n2)
    +    >>> data = [(i, j, sim(points[i], points[j])) for i in range(1, n) for j in range(0, i)]
    +    >>> rdd = sc.parallelize(data, 2)
    +    >>> schema = StructType([StructField("src", LongType(), False), \
    +                 StructField("dst", LongType(),  True), \
    +                 StructField("weight", DoubleType(), True)])
    +    >>> df = spark.createDataFrame(rdd, schema)
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> result = sorted(assignments.collect(), key=lambda x: x.id)
    +    >>> result[0].cluster == result[1].cluster == result[2].cluster == result[3].cluster
    +    True
    +    >>> result[4].cluster == result[5].cluster == result[6].cluster == result[7].cluster
    +    True
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    +    >>> result2 = sorted(assignments2.collect(), key=lambda x: x.id)
    +    >>> result2[0].cluster == result2[1].cluster == result2[2].cluster == result2[3].cluster
    +    True
    +    >>> result2[4].cluster == result2[5].cluster == result2[6].cluster == result2[7].cluster
    +    True
    --- End diff --
    
    Thanks for the comments. Will make changes. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194247139
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,213 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use :py:func:`assignClusters` method
    +    to run the PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +   >>> data = [((long)(1), (long)(0), 0.5), \
    +               ((long)(2), (long)(0), 0.5), \
    +               ((long)(2), (long)(1), 0.7), \
    +               ((long)(3), (long)(0), 0.5), \
    +               ((long)(3), (long)(1), 0.7), \
    +               ((long)(3), (long)(2), 0.9), \
    +               ((long)(4), (long)(0), 0.5), \
    +               ((long)(4), (long)(1), 0.7), \
    +               ((long)(4), (long)(2), 0.9), \
    +               ((long)(4), (long)(3), 1.1), \
    +               ((long)(5), (long)(0), 0.5), \
    +               ((long)(5), (long)(1), 0.7), \
    +               ((long)(5), (long)(2), 0.9), \
    +               ((long)(5), (long)(3), 1.1), \
    +               ((long)(5), (long)(4), 1.3)]
    +    >>> df = spark.createDataFrame(data).toDF("src", "dst", "weight")
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> assignments.sort(assignments.id).show(truncate=False)
    +    +---+-------+
    +    |id |cluster|
    +    +---+-------+
    +    |0  |1      |
    +    |1  |1      |
    +    |2  |1      |
    +    |3  |1      |
    +    |4  |1      |
    +    |5  |0      |
    +    +---+-------+
    +    ...
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    --- End diff --
    
    Will remove line 1214 and the following.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91628/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194214516
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,204 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    +    PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +    >>> from pyspark.sql.types import DoubleType, LongType, StructField, StructType
    +    >>> import math
    +    >>> def genCircle(r, n):
    +    ...     points = []
    +    ...     for i in range(0, n):
    +    ...         theta = 2.0 * math.pi * i / n
    +    ...         points.append((r * math.cos(theta), r * math.sin(theta)))
    +    ...     return points
    +    >>> def sim(x, y):
    +    ...     dist = (x[0] - y[0]) * (x[0] - y[0]) + (x[1] - y[1]) * (x[1] - y[1])
    +    ...     return math.exp(-dist / 2.0)
    +    >>> r1 = 1.0
    +    >>> n1 = 10
    +    >>> r2 = 4.0
    +    >>> n2 = 40
    +    >>> n = n1 + n2
    +    >>> points = genCircle(r1, n1) + genCircle(r2, n2)
    +    >>> data = [(i, j, sim(points[i], points[j])) for i in range(1, n) for j in range(0, i)]
    +    >>> rdd = sc.parallelize(data, 2)
    +    >>> schema = StructType([StructField("src", LongType(), False), \
    +                 StructField("dst", LongType(),  True), \
    +                 StructField("weight", DoubleType(), True)])
    +    >>> df = spark.createDataFrame(rdd, schema)
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> result = sorted(assignments.collect(), key=lambda x: x.id)
    +    >>> result[0].cluster == result[1].cluster == result[2].cluster == result[3].cluster
    +    True
    +    >>> result[4].cluster == result[5].cluster == result[6].cluster == result[7].cluster
    +    True
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    +    >>> result2 = sorted(assignments2.collect(), key=lambda x: x.id)
    +    >>> result2[0].cluster == result2[1].cluster == result2[2].cluster == result2[3].cluster
    +    True
    +    >>> result2[4].cluster == result2[5].cluster == result2[6].cluster == result2[7].cluster
    +    True
    +    >>> pic3 = PowerIterationClustering(k=4, initMode="degree", srcCol="source", dstCol="dest")
    +    >>> pic3.getSrcCol()
    +    'source'
    +    >>> pic3.getDstCol()
    +    'dest'
    +    >>> pic3.getK()
    +    4
    +    >>> pic3.getMaxIter()
    +    20
    +    >>> pic3.getInitMode()
    +    'degree'
    +
    +    .. versionadded:: 2.4.0
    +    """
    +
    +    k = Param(Params._dummy(), "k",
    +              "The number of clusters to create. Must be > 1.",
    +              typeConverter=TypeConverters.toInt)
    +    initMode = Param(Params._dummy(), "initMode",
    +                     "The initialization algorithm. This can be either " +
    +                     "'random' to use a random vector as vertex properties, or 'degree' to use " +
    +                     "a normalized sum of similarities with other vertices.  Supported options: " +
    +                     "'random' and 'degree'.",
    +                     typeConverter=TypeConverters.toString)
    +    srcCol = Param(Params._dummy(), "srcCol",
    +                   "Name of the input column for source vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +    dstCol = Param(Params._dummy(), "dstCol",
    +                   "Name of the input column for destination vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +
    +    @keyword_only
    +    def __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                 weightCol=None):
    +        """
    +        __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                 weightCol=None)
    +        """
    +        super(PowerIterationClustering, self).__init__()
    +        self._java_obj = self._new_java_obj(
    +            "org.apache.spark.ml.clustering.PowerIterationClustering", self.uid)
    +        self._setDefault(k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst")
    +        kwargs = self._input_kwargs
    +        self.setParams(**kwargs)
    +
    +    @keyword_only
    +    @since("2.4.0")
    +    def setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                  weightCol=None):
    +        """
    +        setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                  weightCol=None)
    +        Sets params for PowerIterationClustering.
    +        """
    +        kwargs = self._input_kwargs
    +        return self._set(**kwargs)
    +
    +    @since("2.4.0")
    +    def setK(self, value):
    +        """
    +        Sets the value of :py:attr:`k`.
    +        """
    +        return self._set(k=value)
    +
    +    @since("2.4.0")
    +    def getK(self):
    +        """
    +        Gets the value of :py:attr:`k`.
    +        """
    +        return self.getOrDefault(self.k)
    +
    +    @since("2.4.0")
    +    def setInitMode(self, value):
    +        """
    +        Sets the value of :py:attr:`initMode`.
    +        """
    +        return self._set(initMode=value)
    +
    +    @since("2.4.0")
    +    def getInitMode(self):
    +        """
    +        Gets the value of `initMode`
    +        """
    +        return self.getOrDefault(self.initMode)
    +
    +    @since("2.4.0")
    +    def setSrcCol(self, value):
    +        """
    +        Sets the value of :py:attr:`srcCol`.
    +        """
    +        return self._set(srcCol=value)
    +
    +    @since("2.4.0")
    +    def getSrcCol(self):
    +        """
    +        Gets the value of :py:attr:`srcCol`.
    +        """
    +        return self.getOrDefault(self.srcCol)
    +
    +    @since("2.4.0")
    +    def setDstCol(self, value):
    +        """
    +        Sets the value of :py:attr:`dstCol`.
    +        """
    +        return self._set(dstCol=value)
    +
    +    @since("2.4.0")
    +    def getDstCol(self):
    +        """
    +        Gets the value of :py:attr:`dstCol`.
    +        """
    +        return self.getOrDefault(self.dstCol)
    +
    +    @since("2.4.0")
    +    def assignClusters(self, dataset):
    +        """
    +        Run the PIC algorithm and returns a cluster assignment for each input vertex.
    +
    +        :param dataset:
    +          A dataset with columns src, dst, weight representing the affinity matrix,
    +          which is the matrix A in the PIC paper. Suppose the src column value is i,
    +          the dst column value is j, the weight column value is similarity s,,ij,,
    +          which must be nonnegative. This is a symmetric matrix and hence
    +          s,,ij,, = s,,ji,,. For any (i, j) with nonzero similarity, there should be
    +          either (i, j, s,,ij,,) or (j, i, s,,ji,,) in the input. Rows with i = j are
    +          ignored, because we assume s,,ij,, = 0.0.
    +
    +        :return:
    +          A dataset that contains columns of vertex id and the corresponding cluster for
    +          the id. The schema of it will be:
    +          - id: Long
    +          - cluster: Int
    --- End diff --
    
    add `.. versionadded:: 2.4.0`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91578/testReport)** for PR 21513 at commit [`cdc97c1`](https://github.com/apache/spark/commit/cdc97c103ea3f3da60e5e93482a7994f94e790c8).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    LGTM. Thanks! @mengxr Would you mind take a look ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91623/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194214535
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,204 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    --- End diff --
    
    ```
    ... use :py:func:`assignClusters` method ...
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21513


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91578/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91581/testReport)** for PR 21513 at commit [`2d3c760`](https://github.com/apache/spark/commit/2d3c760607388facacf11974d3c3615bb6db0c41).
     * This patch **fails Python style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91581/testReport)** for PR 21513 at commit [`2d3c760`](https://github.com/apache/spark/commit/2d3c760607388facacf11974d3c3615bb6db0c41).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3879/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91578/testReport)** for PR 21513 at commit [`cdc97c1`](https://github.com/apache/spark/commit/cdc97c103ea3f3da60e5e93482a7994f94e790c8).
     * This patch **fails Python style tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    @mengxr @WeichenXu123 Could you please review? Thanks a lot in advance!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194167552
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1159,216 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    +    PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +    >>> from pyspark.sql.types import StructField, StructType
    +    >>> import math
    +    >>> def genCircle(r, n):
    +    ...     points = []
    +    ...     for i in range(0, n):
    +    ...         theta = 2.0 * math.pi * i / n
    +    ...         points.append((r * math.cos(theta), r * math.sin(theta)))
    +    ...     return points
    +    >>> def sim(x, y):
    +    ...     dist = (x[0] - y[0]) * (x[0] - y[0]) + (x[1] - y[1]) * (x[1] - y[1])
    +    ...     return math.exp(-dist / 2.0)
    +    >>> r1 = 1.0
    +    >>> n1 = 10
    +    >>> r2 = 4.0
    +    >>> n2 = 40
    +    >>> n = n1 + n2
    +    >>> points = genCircle(r1, n1) + genCircle(r2, n2)
    +    >>> data = [(i, j, sim(points[i], points[j])) for i in range(1, n) for j in range(0, i)]
    +    >>> rdd = sc.parallelize(data, 2)
    +    >>> schema = StructType([StructField("src", LongType(), False), \
    +                 StructField("dst", LongType(),  True), \
    +                 StructField("weight", DoubleType(), True)])
    +    >>> df = spark.createDataFrame(rdd, schema)
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> result = sorted(assignments.collect(), key=lambda x: x.id)
    +    >>> result[0].cluster == result[1].cluster == result[2].cluster == result[3].cluster
    +    True
    +    >>> result[4].cluster == result[5].cluster == result[6].cluster == result[7].cluster
    +    True
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    +    >>> result2 = sorted(assignments2.collect(), key=lambda x: x.id)
    +    >>> result2[0].cluster == result2[1].cluster == result2[2].cluster == result2[3].cluster
    +    True
    +    >>> result2[4].cluster == result2[5].cluster == result2[6].cluster == result2[7].cluster
    +    True
    +    >>> pic3 = PowerIterationClustering(k=4, initMode="degree", srcCol="source", dstCol="dest")
    +    >>> pic3.getSrcCol()
    +    'source'
    +    >>> pic3.getDstCol()
    +    'dest'
    +    >>> pic3.getK()
    +    4
    +    >>> pic3.getMaxIter()
    +    20
    +    >>> pic3.getInitMode()
    +    'degree'
    +
    +    .. versionadded:: 2.4.0
    +    """
    +
    +    k = Param(Params._dummy(), "k",
    +              "The number of clusters to create. Must be > 1.",
    +              typeConverter=TypeConverters.toInt)
    +    initMode = Param(Params._dummy(), "initMode",
    +                     "The initialization algorithm. This can be either " +
    +                     "'random' to use a random vector as vertex properties, or 'degree' to use " +
    +                     "a normalized sum of similarities with other vertices.  Supported options: " +
    +                     "'random' and 'degree'.",
    +                     typeConverter=TypeConverters.toString)
    +    srcCol = Param(Params._dummy(), "srcCol",
    +                   "Name of the input column for source vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +    dstCol = Param(Params._dummy(), "dstCol",
    +                   "Name of the input column for destination vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +
    +    @keyword_only
    +    def __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                 weightCol=None):
    +        """
    +        __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                 weightCol=None)
    +        """
    +        super(PowerIterationClustering, self).__init__()
    +        self._java_obj = self._new_java_obj(
    +            "org.apache.spark.ml.clustering.PowerIterationClustering", self.uid)
    +        self._setDefault(k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst")
    +        kwargs = self._input_kwargs
    +        self.setParams(**kwargs)
    +
    +    @keyword_only
    +    @since("2.4.0")
    +    def setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                  weightCol=None):
    +        """
    +        setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                  weightCol=None)
    +        Sets params for PowerIterationClustering.
    +        """
    +        kwargs = self._input_kwargs
    +        return self._set(**kwargs)
    +
    +    @since("2.4.0")
    +    def setK(self, value):
    +        """
    +        Sets the value of :py:attr:`k`.
    +        """
    +        return self._set(k=value)
    +
    +    @since("2.4.0")
    +    def getK(self):
    +        """
    +        Gets the value of :py:attr:`k`.
    +        """
    +        return self.getOrDefault(self.k)
    +
    +    @since("2.4.0")
    +    def setInitMode(self, value):
    +        """
    +        Sets the value of :py:attr:`initMode`.
    +        """
    +        return self._set(initMode=value)
    +
    +    @since("2.4.0")
    +    def getInitMode(self):
    +        """
    +        Gets the value of `initMode`
    +        """
    +        return self.getOrDefault(self.initMode)
    +
    +    @since("2.4.0")
    +    def setSrcCol(self, value):
    +        """
    +        Sets the value of :py:attr:`srcCol`.
    +        """
    +        return self._set(srcCol=value)
    +
    +    @since("2.4.0")
    +    def getSrcCol(self):
    +        """
    +        Gets the value of :py:attr:`srcCol`.
    +        """
    +        return self.getOrDefault(self.srcCol)
    +
    +    @since("2.4.0")
    +    def setDstCol(self, value):
    +        """
    +        Sets the value of :py:attr:`dstCol`.
    +        """
    +        return self._set(dstCol=value)
    +
    +    @since("2.4.0")
    +    def getDstCol(self):
    +        """
    +        Gets the value of :py:attr:`dstCol`.
    +        """
    +        return self.getOrDefault(self.dstCol)
    +
    +    @since("2.4.0")
    +    def assignClusters(self, dataset):
    +        """
    +        Run the PIC algorithm and returns a cluster assignment for each input vertex.
    +
    +        :param dataset:
    +          A dataset with columns src, dst, weight representing the affinity matrix,
    +          which is the matrix A in the PIC paper. Suppose the src column value is i,
    +          the dst column value is j, the weight column value is similarity s,,ij,,
    +          which must be nonnegative. This is a symmetric matrix and hence
    +          s,,ij,, = s,,ji,,. For any (i, j) with nonzero similarity, there should be
    +          either (i, j, s,,ij,,) or (j, i, s,,ji,,) in the input. Rows with i = j are
    +          ignored, because we assume s,,ij,, = 0.0.
    +
    +        :return: A dataset that contains columns of vertex id and the corresponding cluster for
    +                 the id. The schema of it will be:
    +                   - id: Long
    +                   - cluster: Int
    +        """
    +        weightCol = None
    +        w = None
    +        if (self.isDefined(self.weightCol)):
    +            weightCol = self.getWeightCol()
    +        if (weightCol is None or len(weightCol) == 0):
    +            w = lit(1.0)
    +        else:
    +            w = col(weightCol).cast(DoubleType())
    +        srcCol = self.getSrcCol()
    +        dstCol = self.getDstCol()
    +        df = dataset.select(col(srcCol).cast(LongType()), col(dstCol).cast(LongType()), w)
    +        data = df.rdd.map(lambda x: (x[0], x[1], x[2]))
    +        algorithm = MLlibPowerIterationClustering()
    --- End diff --
    
    I think here should wrap `ml.clustering.PowerIterationClustering`, not the one in `pyspark.mllib`. So that it can make sure behavior consistency with scala api.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3863/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194214831
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,204 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    +    PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +    >>> from pyspark.sql.types import DoubleType, LongType, StructField, StructType
    +    >>> import math
    +    >>> def genCircle(r, n):
    +    ...     points = []
    +    ...     for i in range(0, n):
    +    ...         theta = 2.0 * math.pi * i / n
    +    ...         points.append((r * math.cos(theta), r * math.sin(theta)))
    +    ...     return points
    +    >>> def sim(x, y):
    +    ...     dist = (x[0] - y[0]) * (x[0] - y[0]) + (x[1] - y[1]) * (x[1] - y[1])
    +    ...     return math.exp(-dist / 2.0)
    +    >>> r1 = 1.0
    +    >>> n1 = 10
    +    >>> r2 = 4.0
    +    >>> n2 = 40
    +    >>> n = n1 + n2
    +    >>> points = genCircle(r1, n1) + genCircle(r2, n2)
    +    >>> data = [(i, j, sim(points[i], points[j])) for i in range(1, n) for j in range(0, i)]
    +    >>> rdd = sc.parallelize(data, 2)
    +    >>> schema = StructType([StructField("src", LongType(), False), \
    +                 StructField("dst", LongType(),  True), \
    +                 StructField("weight", DoubleType(), True)])
    +    >>> df = spark.createDataFrame(rdd, schema)
    --- End diff --
    
    The test code here is too complex to be in doctest. could you change it to code like:
    ``
    df = sc.parallelize(...).toDF(...)
    ``
    generate a small, hardcoded dataset.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91584/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3860/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91584 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91584/testReport)** for PR 21513 at commit [`6a35dde`](https://github.com/apache/spark/commit/6a35dde8592a476aa7c5334ee0cf1494fc5bc60a).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91628/testReport)** for PR 21513 at commit [`7c07725`](https://github.com/apache/spark/commit/7c077252570b2c2458bacff06194a3943611fc7c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194215008
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,204 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    +    PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +    >>> from pyspark.sql.types import DoubleType, LongType, StructField, StructType
    +    >>> import math
    +    >>> def genCircle(r, n):
    +    ...     points = []
    +    ...     for i in range(0, n):
    +    ...         theta = 2.0 * math.pi * i / n
    +    ...         points.append((r * math.cos(theta), r * math.sin(theta)))
    +    ...     return points
    +    >>> def sim(x, y):
    +    ...     dist = (x[0] - y[0]) * (x[0] - y[0]) + (x[1] - y[1]) * (x[1] - y[1])
    +    ...     return math.exp(-dist / 2.0)
    +    >>> r1 = 1.0
    +    >>> n1 = 10
    +    >>> r2 = 4.0
    +    >>> n2 = 40
    +    >>> n = n1 + n2
    +    >>> points = genCircle(r1, n1) + genCircle(r2, n2)
    +    >>> data = [(i, j, sim(points[i], points[j])) for i in range(1, n) for j in range(0, i)]
    +    >>> rdd = sc.parallelize(data, 2)
    +    >>> schema = StructType([StructField("src", LongType(), False), \
    +                 StructField("dst", LongType(),  True), \
    +                 StructField("weight", DoubleType(), True)])
    +    >>> df = spark.createDataFrame(rdd, schema)
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> result = sorted(assignments.collect(), key=lambda x: x.id)
    +    >>> result[0].cluster == result[1].cluster == result[2].cluster == result[3].cluster
    +    True
    +    >>> result[4].cluster == result[5].cluster == result[6].cluster == result[7].cluster
    +    True
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    +    >>> result2 = sorted(assignments2.collect(), key=lambda x: x.id)
    +    >>> result2[0].cluster == result2[1].cluster == result2[2].cluster == result2[3].cluster
    +    True
    +    >>> result2[4].cluster == result2[5].cluster == result2[6].cluster == result2[7].cluster
    +    True
    --- End diff --
    
    Let's use a simpler way to check result, like:
    ```
    >>> assignments.sort(assignments.id).show(truncate=False)
    ...
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194244592
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,213 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use :py:func:`assignClusters` method
    +    to run the PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +   >>> data = [((long)(1), (long)(0), 0.5), \
    --- End diff --
    
    Users do not know that we make `long` an alias of `int` in PySpark + Python 3.
    
    I think in both Py2 and 3, PySpark infers Python int/long as long type in DataFrame. Could you help verify? If that is the case, we can drop `(long)(...)` here. If not, we can cast the columns to long type after creating the DataFrame.
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91581/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91623/testReport)** for PR 21513 at commit [`fcb9a51`](https://github.com/apache/spark/commit/fcb9a51e0cb07d0c77c98d93797b21a1b670aa41).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91628/testReport)** for PR 21513 at commit [`7c07725`](https://github.com/apache/spark/commit/7c077252570b2c2458bacff06194a3943611fc7c).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by WeichenXu123 <gi...@git.apache.org>.
Github user WeichenXu123 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194214431
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,204 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use `assignClusters` method to run the
    +    PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +    >>> from pyspark.sql.types import DoubleType, LongType, StructField, StructType
    +    >>> import math
    +    >>> def genCircle(r, n):
    +    ...     points = []
    +    ...     for i in range(0, n):
    +    ...         theta = 2.0 * math.pi * i / n
    +    ...         points.append((r * math.cos(theta), r * math.sin(theta)))
    +    ...     return points
    +    >>> def sim(x, y):
    +    ...     dist = (x[0] - y[0]) * (x[0] - y[0]) + (x[1] - y[1]) * (x[1] - y[1])
    +    ...     return math.exp(-dist / 2.0)
    +    >>> r1 = 1.0
    +    >>> n1 = 10
    +    >>> r2 = 4.0
    +    >>> n2 = 40
    +    >>> n = n1 + n2
    +    >>> points = genCircle(r1, n1) + genCircle(r2, n2)
    +    >>> data = [(i, j, sim(points[i], points[j])) for i in range(1, n) for j in range(0, i)]
    +    >>> rdd = sc.parallelize(data, 2)
    +    >>> schema = StructType([StructField("src", LongType(), False), \
    +                 StructField("dst", LongType(),  True), \
    +                 StructField("weight", DoubleType(), True)])
    +    >>> df = spark.createDataFrame(rdd, schema)
    +    >>> pic = PowerIterationClustering()
    +    >>> assignments = pic.setK(2).setMaxIter(40).setWeightCol("weight").assignClusters(df)
    +    >>> result = sorted(assignments.collect(), key=lambda x: x.id)
    +    >>> result[0].cluster == result[1].cluster == result[2].cluster == result[3].cluster
    +    True
    +    >>> result[4].cluster == result[5].cluster == result[6].cluster == result[7].cluster
    +    True
    +    >>> pic_path = temp_path + "/pic"
    +    >>> pic.save(pic_path)
    +    >>> pic2 = PowerIterationClustering.load(pic_path)
    +    >>> pic2.getK()
    +    2
    +    >>> pic2.getMaxIter()
    +    40
    +    >>> assignments2 = pic2.assignClusters(df)
    +    >>> result2 = sorted(assignments2.collect(), key=lambda x: x.id)
    +    >>> result2[0].cluster == result2[1].cluster == result2[2].cluster == result2[3].cluster
    +    True
    +    >>> result2[4].cluster == result2[5].cluster == result2[6].cluster == result2[7].cluster
    +    True
    +    >>> pic3 = PowerIterationClustering(k=4, initMode="degree", srcCol="source", dstCol="dest")
    +    >>> pic3.getSrcCol()
    +    'source'
    +    >>> pic3.getDstCol()
    +    'dest'
    +    >>> pic3.getK()
    +    4
    +    >>> pic3.getMaxIter()
    +    20
    +    >>> pic3.getInitMode()
    +    'degree'
    +
    +    .. versionadded:: 2.4.0
    +    """
    +
    +    k = Param(Params._dummy(), "k",
    +              "The number of clusters to create. Must be > 1.",
    +              typeConverter=TypeConverters.toInt)
    +    initMode = Param(Params._dummy(), "initMode",
    +                     "The initialization algorithm. This can be either " +
    +                     "'random' to use a random vector as vertex properties, or 'degree' to use " +
    +                     "a normalized sum of similarities with other vertices.  Supported options: " +
    +                     "'random' and 'degree'.",
    +                     typeConverter=TypeConverters.toString)
    +    srcCol = Param(Params._dummy(), "srcCol",
    +                   "Name of the input column for source vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +    dstCol = Param(Params._dummy(), "dstCol",
    +                   "Name of the input column for destination vertex IDs.",
    +                   typeConverter=TypeConverters.toString)
    +
    +    @keyword_only
    +    def __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                 weightCol=None):
    +        """
    +        __init__(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                 weightCol=None)
    +        """
    +        super(PowerIterationClustering, self).__init__()
    +        self._java_obj = self._new_java_obj(
    +            "org.apache.spark.ml.clustering.PowerIterationClustering", self.uid)
    +        self._setDefault(k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst")
    +        kwargs = self._input_kwargs
    +        self.setParams(**kwargs)
    +
    +    @keyword_only
    +    @since("2.4.0")
    +    def setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",
    +                  weightCol=None):
    +        """
    +        setParams(self, k=2, maxIter=20, initMode="random", srcCol="src", dstCol="dst",\
    +                  weightCol=None)
    +        Sets params for PowerIterationClustering.
    +        """
    +        kwargs = self._input_kwargs
    +        return self._set(**kwargs)
    +
    +    @since("2.4.0")
    +    def setK(self, value):
    +        """
    +        Sets the value of :py:attr:`k`.
    +        """
    +        return self._set(k=value)
    +
    +    @since("2.4.0")
    +    def getK(self):
    +        """
    +        Gets the value of :py:attr:`k`.
    --- End diff --
    
    change to
    ```
    Gets the value of :py:attr:`k` or its default value.
    ```
    here and every other places.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91623/testReport)** for PR 21513 at commit [`fcb9a51`](https://github.com/apache/spark/commit/fcb9a51e0cb07d0c77c98d93797b21a1b670aa41).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21513#discussion_r194247137
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -1156,6 +1157,213 @@ def getKeepLastCheckpoint(self):
             return self.getOrDefault(self.keepLastCheckpoint)
     
     
    +@inherit_doc
    +class PowerIterationClustering(HasMaxIter, HasWeightCol, JavaParams, JavaMLReadable,
    +                               JavaMLWritable):
    +    """
    +    .. note:: Experimental
    +
    +    Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by
    +    <a href=http://www.icml2010.org/papers/387.pdf>Lin and Cohen</a>. From the abstract:
    +    PIC finds a very low-dimensional embedding of a dataset using truncated power
    +    iteration on a normalized pair-wise similarity matrix of the data.
    +
    +    This class is not yet an Estimator/Transformer, use :py:func:`assignClusters` method
    +    to run the PowerIterationClustering algorithm.
    +
    +    .. seealso:: `Wikipedia on Spectral clustering \
    +    <http://en.wikipedia.org/wiki/Spectral_clustering>`_
    +
    +   >>> data = [((long)(1), (long)(0), 0.5), \
    --- End diff --
    
    @mengxr Thank you very much for your review. 
    Seems to me that only Python 3 infers int/long as long type in DataFrame. 
    ```
    if sys.version >= '3':
        basestring = unicode = str
        long = int
    ```
    But I should remove the (long) casting because the PIC scala code cast the columns to long type. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    LGTM. Merged into master. Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3881/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    **[Test build #91584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91584/testReport)** for PR 21513 at commit [`6a35dde`](https://github.com/apache/spark/commit/6a35dde8592a476aa7c5334ee0cf1494fc5bc60a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21513
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org