You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by WeichenXu123 <gi...@git.apache.org> on 2018/01/09 18:52:00 UTC

[GitHub] spark pull request #20209: [SPARK-23008][ML] OnehotEncoderEstimator python A...

GitHub user WeichenXu123 opened a pull request:

    https://github.com/apache/spark/pull/20209

    [SPARK-23008][ML] OnehotEncoderEstimator python API

    ## What changes were proposed in this pull request?
    
    OnehotEncoderEstimator python API.
    
    ## How was this patch tested?
    
    doctest

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/WeichenXu123/spark ohe_py

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20209.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20209
    
----
commit f6215fc45901456dea8a4fb32f7c87907bb2fbfb
Author: WeichenXu <we...@...>
Date:   2018-01-09T01:26:14Z

    init pr

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    **[Test build #85866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85866/testReport)** for PR 20209 at commit [`f6215fc`](https://github.com/apache/spark/commit/f6215fc45901456dea8a4fb32f7c87907bb2fbfb).
     * This patch **fails Python style tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class OneHotEncoderEstimator(JavaEstimator, HasInputCols, HasOutputCols, HasHandleInvalid,`
      * `class OneHotEncoderModel(JavaModel, JavaMLReadable, JavaMLWritable):`
      * `class HasOutputCols(Params):`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20209: [SPARK-23008][ML] OnehotEncoderEstimator python A...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20209


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85866/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86001/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20209: [SPARK-23008][ML] OnehotEncoderEstimator python A...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20209#discussion_r161110909
  
    --- Diff: python/pyspark/ml/feature.py ---
    @@ -1641,6 +1642,118 @@ def getDropLast(self):
             return self.getOrDefault(self.dropLast)
     
     
    +@inherit_doc
    +class OneHotEncoderEstimator(JavaEstimator, HasInputCols, HasOutputCols, HasHandleInvalid,
    +                             JavaMLReadable, JavaMLWritable):
    +    """
    +    A one-hot encoder that maps a column of category indices to a column of binary vectors, with
    +    at most a single one-value per row that indicates the input category index.
    +    For example with 5 categories, an input value of 2.0 would map to an output vector of
    +    `[0.0, 0.0, 1.0, 0.0]`.
    +    The last category is not included by default (configurable via `dropLast`),
    +    because it makes the vector entries sum up to one, and hence linearly dependent.
    +    So an input value of 4.0 maps to `[0.0, 0.0, 0.0, 0.0]`.
    +
    +    Note: This is different from scikit-learn's OneHotEncoder, which keeps all categories.
    +    The output vectors are sparse.
    +
    +    When `handleInvalid` is configured to 'keep', an extra "category" indicating invalid values is
    +    added as last category. So when `dropLast` is true, invalid values are encoded as all-zeros
    +    vector.
    +
    +    Note: When encoding multi-column by using `inputCols` and `outputCols` params, input/output
    +    cols come in pairs, specified by the order in the arrays, and each pair is treated
    +    independently.
    +
    +    See `StringIndexer` for converting categorical values into category indices
    +
    +    >>> from pyspark.ml.linalg import Vectors
    +    >>> df = spark.createDataFrame([(0.0,), (1.0,), (2.0,)], ["input"])
    +    >>> ohe = OneHotEncoderEstimator(inputCols=["input"], outputCols=["output"])
    +    >>> model = ohe.fit(df)
    +    >>> model.transform(df).head().output
    +    SparseVector(2, {0: 1.0})
    +    >>> ohePath = temp_path + "/oheEstimator"
    +    >>> ohe.save(ohePath)
    +    >>> loadedOHE = OneHotEncoderEstimator.load(ohePath)
    +    >>> loadedOHE.getInputCols() == ohe.getInputCols()
    +    True
    +    >>> modelPath = temp_path + "/ohe-model"
    +    >>> model.save(modelPath)
    +    >>> loadedModel = OneHotEncoderModel.load(modelPath)
    +    >>> loadedModel.categorySizes == model.categorySizes
    +    True
    +
    +    .. versionadded:: 2.3.0
    +    """
    +
    +    handleInvalid = Param(Params._dummy(), "handleInvalid", "How to handle invalid data during " +
    +                          "transform(). Options are 'keep' (invalid data presented as an extra " +
    +                          "categorical feature) or error (throw an error). Note that this Param " +
    +                          "is only used during transform; during fitting, invalid data will " +
    +                          "result in an error.",
    +                          typeConverter=TypeConverters.toString)
    +
    +    dropLast = Param(Params._dummy(), "dropLast", "whether to drop the last category",
    +                     typeConverter=TypeConverters.toBoolean)
    +
    +    @keyword_only
    +    def __init__(self, inputCols=None, outputCols=None, handleInvalid="error", dropLast=True):
    +        """
    +        __init__(self, inputCols=None, outputCols=None, handleInvalid="error", dropLast=True)
    +        """
    +        super(OneHotEncoderEstimator, self).__init__()
    +        self._java_obj = self._new_java_obj(
    +            "org.apache.spark.ml.feature.OneHotEncoderEstimator", self.uid)
    +        self._setDefault(handleInvalid="error", dropLast=True)
    +        kwargs = self._input_kwargs
    +        self.setParams(**kwargs)
    +
    +    @keyword_only
    +    @since("1.4.0")
    --- End diff --
    
    2.3.0


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    **[Test build #85871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85871/testReport)** for PR 20209 at commit [`2c10416`](https://github.com/apache/spark/commit/2c10416a8a06f4c574dc662a1d7bb7dbcdd36a37).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    I'll review this now


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    **[Test build #86001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86001/testReport)** for PR 20209 at commit [`c7db798`](https://github.com/apache/spark/commit/c7db798d7faf343a27c778fd48f2cdd77d732107).
     * This patch **fails from timeout after a configured wait of \`250m\`**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    **[Test build #86001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86001/testReport)** for PR 20209 at commit [`c7db798`](https://github.com/apache/spark/commit/c7db798d7faf343a27c778fd48f2cdd77d732107).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    Thanks @WeichenXu123 !  LGTM
    As I just mentioned in the JIRA, I'm going to backport this to branch-2.3 for release with 2.3.0 since this is arguably a bug fix for ML in streaming.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    **[Test build #85871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85871/testReport)** for PR 20209 at commit [`2c10416`](https://github.com/apache/spark/commit/2c10416a8a06f4c574dc662a1d7bb7dbcdd36a37).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85871/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20209: [SPARK-23008][ML] OnehotEncoderEstimator python API

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20209
  
    **[Test build #85866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85866/testReport)** for PR 20209 at commit [`f6215fc`](https://github.com/apache/spark/commit/f6215fc45901456dea8a4fb32f7c87907bb2fbfb).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org