You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/19 01:08:45 UTC
[GitHub] [spark] huaxingao opened a new pull request #27954:
[SPARK-31885][ML] Implement VarianceThresholdSelector
huaxingao opened a new pull request #27954: [SPARK-31885][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954
### What changes were proposed in this pull request?
Implement a Feature selector that removes all low-variance features. Features with a
variance lower than the threshold will be removed. The default is to keep all features with non-zero variance, i.e. remove the features that have the same value in all samples.
### Why are the changes needed?
VarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. The idea is when a feature doesn’t vary much within itself, it generally has very little predictive power.
scikit has implemented this selector.
https://scikit-learn.org/stable/modules/feature_selection.html#variance-threshold
### Does this PR introduce any user-facing change?
Yes.
### How was this patch tested?
Add new test suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600935926
**[Test build #120015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120015/testReport)** for PR 27954 at commit [`5016cf9`](https://github.com/apache/spark/commit/5016cf96eff6d1ff7ba5f9f6e89949d88c821904).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600936661
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24733/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601841635
**[Test build #120113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120113/testReport)** for PR 27954 at commit [`886b2dc`](https://github.com/apache/spark/commit/886b2dca6064118912180d3c511d857963683eeb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600977511
**[Test build #120022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120022/testReport)** for PR 27954 at commit [`1455987`](https://github.com/apache/spark/commit/1455987ef350cad1a190d8f1c26ad553fe8b2751).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601838619
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24826/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600996889
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120022/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r395955515
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than or equal to this threshold
+ * will be removed. The default value is 0.0.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed. The default value is 0.0.", ParamValidators.gtEq(0))
+ setDefault(varianceThreshold -> 0.0)
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
Review comment:
'lower' -> 'not greater than'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-602004473
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600954866
**[Test build #120015 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120015/testReport)** for PR 27954 at commit [`5016cf9`](https://github.com/apache/spark/commit/5016cf96eff6d1ff7ba5f9f6e89949d88c821904).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601838615
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601885682
**[Test build #120113 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120113/testReport)** for PR 27954 at commit [`886b2dc`](https://github.com/apache/spark/commit/886b2dca6064118912180d3c511d857963683eeb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600996889
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120022/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600996537
**[Test build #120022 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120022/testReport)** for PR 27954 at commit [`1455987`](https://github.com/apache/spark/commit/1455987ef350cad1a190d8f1c26ad553fe8b2751).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600996881
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601886230
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120113/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-602147344
Merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600977802
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r394736366
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than this threshold will
+ * be removed.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed.", ParamValidators.gt(0))
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ var Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
+ // if varianceThreshold not set, remove the features that have the same value in all samples.
+ val features = if (!isSet(varianceThreshold)) {
+ // use max and min to avoid numeric precision issues for constant features
+ result.filter { case (((vari, max), min), _) => ((max != min) && (vari != 0)) }
Review comment:
I follow scikit-learn implementation to use max and min to avoid numeric precision issues for constant features.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600977511
**[Test build #120022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120022/testReport)** for PR 27954 at commit [`1455987`](https://github.com/apache/spark/commit/1455987ef350cad1a190d8f1c26ad553fe8b2751).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-602004473
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600955154
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601838619
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24826/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601998807
**[Test build #120121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120121/testReport)** for PR 27954 at commit [`3f5ecef`](https://github.com/apache/spark/commit/3f5ecef0988f2a1838c407fb65876ff65847e7ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r394890414
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than this threshold will
+ * be removed.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed.", ParamValidators.gt(0))
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ val Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
+ // if varianceThreshold not set, remove the features that have the same value in all samples.
+ val features = if (!isSet(varianceThreshold)) {
+ // use max and min to avoid numeric precision issues for constant features
+ result.filter { case (((vari, max), min), _) => ((max != min) && (vari != 0)) }
+ } else {
+ result.filter { case (((vari, _), _), _) => !(vari < getVarianceThreshold) }
+ }
+
+ val indices = features.map { case (((_, _), _), index) => index }
Review comment:
we can simplify this logic like:
```scala
val numFeatures = max.size
val indices = Array.tabulate(numFeatures) { i =>
(i, if (max(i) == min(i)) 0.0 else variance(i))
}.filter(_._2 >= getVarianceThreshold).map(_._1)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600935926
**[Test build #120015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120015/testReport)** for PR 27954 at commit [`5016cf9`](https://github.com/apache/spark/commit/5016cf96eff6d1ff7ba5f9f6e89949d88c821904).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600977802
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600936661
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24733/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601998894
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24835/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-602004392
**[Test build #120121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120121/testReport)** for PR 27954 at commit [`3f5ecef`](https://github.com/apache/spark/commit/3f5ecef0988f2a1838c407fb65876ff65847e7ad).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r395955676
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -92,16 +93,10 @@ with DefaultParamsWritable {
.select("summary.max", "summary.min", "summary.variance")
.first()
- val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
- // if varianceThreshold not set, remove the features that have the same value in all samples.
- val features = if (!isSet(varianceThreshold)) {
- // use max and min to avoid numeric precision issues for constant features
- result.filter { case (((vari, max), min), _) => ((max != min) && (vari != 0)) }
- } else {
- result.filter { case (((vari, _), _), _) => !(vari < getVarianceThreshold) }
- }
-
- val indices = features.map { case (((_, _), _), index) => index }
+ val numFeatures = maxs.size
+ val indices = Array.tabulate(numFeatures) { i =>
+ (i, if (maxs(i) == mins(i)) 0.0 else variances(i))
+ } .filter(_._2 > getVarianceThreshold).map(_._1)
Review comment:
nit : no space after '}'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r395398439
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than this threshold will
+ * be removed.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed.", ParamValidators.gt(0))
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ val Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
+ // if varianceThreshold not set, remove the features that have the same value in all samples.
+ val features = if (!isSet(varianceThreshold)) {
Review comment:
I thought we keep the features >= threshold, so I can't default it to 0 (variance 0 will be kept too).
Maybe I should change the definition to "keep the features > threshold"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600955160
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120015/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600935687
This Selector has quite some common code with other Selectors. I will refactor all the Selectors in https://github.com/apache/spark/pull/27882.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r395804668
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than this threshold will
+ * be removed.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed.", ParamValidators.gt(0))
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ val Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
+ // if varianceThreshold not set, remove the features that have the same value in all samples.
+ val features = if (!isSet(varianceThreshold)) {
Review comment:
I will change to "keep the features > threshold". I looked sklearn code and did a little test, sklearn removes the <= threshold features.
```
def _get_support_mask(self):
check_is_fitted(self)
return self.variances_ > self.threshold
```
```
>>> data = [[0, 2, 7, 1],
... [1, 4, 9, 5]]
>>> np.var([0, 1])
0.25
>>> np.var([2, 4])
1.0
>>> np.var([7, 9])
1.0
>>> np.var([1, 5])
4.0
>>> VarianceThreshold(threshold=0.2499).fit_transform(data)
array([[0, 2, 7, 1],
[1, 4, 9, 5]])
>>> VarianceThreshold(threshold=0.25).fit_transform(data)
array([[2, 7, 1],
[4, 9, 5]])
>>> VarianceThreshold(threshold=0.9999).fit_transform(data)
array([[2, 7, 1],
[4, 9, 5]])
>>> VarianceThreshold(threshold=1.0).fit_transform(data)
array([[1],
[5]])
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r394891308
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than this threshold will
+ * be removed.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed.", ParamValidators.gt(0))
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ val Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
+ // if varianceThreshold not set, remove the features that have the same value in all samples.
+ val features = if (!isSet(varianceThreshold)) {
+ // use max and min to avoid numeric precision issues for constant features
+ result.filter { case (((vari, max), min), _) => ((max != min) && (vari != 0)) }
+ } else {
+ result.filter { case (((vari, _), _), _) => !(vari < getVarianceThreshold) }
+ }
+
+ val indices = features.map { case (((_, _), _), index) => index }
Review comment:
I guess in sklearn checking `max == min` are just to make variance computation of constant values more stable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601998893
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601998894
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24835/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r394888694
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than this threshold will
+ * be removed.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed.", ParamValidators.gt(0))
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ val Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
+ // if varianceThreshold not set, remove the features that have the same value in all samples.
+ val features = if (!isSet(varianceThreshold)) {
Review comment:
Why not giving `varianceThreshold` a default value = 0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601998807
**[Test build #120121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120121/testReport)** for PR 27954 at commit [`3f5ecef`](https://github.com/apache/spark/commit/3f5ecef0988f2a1838c407fb65876ff65847e7ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601998893
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600977806
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24739/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r395398475
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than this threshold will
+ * be removed.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed.", ParamValidators.gt(0))
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ val Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val result = variances.toArray.zip(maxs.toArray).zip(mins.toArray).zipWithIndex
+ // if varianceThreshold not set, remove the features that have the same value in all samples.
+ val features = if (!isSet(varianceThreshold)) {
+ // use max and min to avoid numeric precision issues for constant features
+ result.filter { case (((vari, max), min), _) => ((max != min) && (vari != 0)) }
+ } else {
+ result.filter { case (((vari, _), _), _) => !(vari < getVarianceThreshold) }
+ }
+
+ val indices = features.map { case (((_, _), _), index) => index }
Review comment:
Yes. Seems it is trying to get around the floating point comparison issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-602004478
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120121/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601886223
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-602004478
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120121/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601886230
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120113/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600936654
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601838615
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600955154
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600936654
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600977806
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24739/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600996881
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601886223
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#discussion_r395955597
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml._
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.stat.Summarizer
+import org.apache.spark.ml.util._
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{StructField, StructType}
+
+
+/**
+ * Params for [[VarianceThresholdSelector]] and [[VarianceThresholdSelectorModel]].
+ */
+private[feature] trait VarianceThresholdSelectorParams extends Params
+ with HasFeaturesCol with HasOutputCol {
+
+ /**
+ * Param for variance threshold. Features with a variance lower than or equal to this threshold
+ * will be removed. The default value is 0.0.
+ *
+ * @group param
+ */
+ @Since("3.1.0")
+ final val varianceThreshold = new DoubleParam(this, "varianceThreshold",
+ "Param for variance threshold. Features with a variance lower than this threshold" +
+ " will be removed. The default value is 0.0.", ParamValidators.gtEq(0))
+ setDefault(varianceThreshold -> 0.0)
+
+ /** @group getParam */
+ @Since("3.1.0")
+ def getVarianceThreshold: Double = $(varianceThreshold)
+}
+
+/**
+ * Feature selector that removes all low-variance features. Features with a
+ * variance lower than the threshold will be removed. The default is to keep
+ * all features with non-zero variance, i.e. remove the features that have the
+ * same value in all samples.
+ */
+@Since("3.1.0")
+final class VarianceThresholdSelector @Since("3.1.0")(@Since("3.1.0") override val uid: String)
+ extends Estimator[VarianceThresholdSelectorModel] with VarianceThresholdSelectorParams
+with DefaultParamsWritable {
+
+ @Since("3.1.0")
+ def this() = this(Identifiable.randomUID("VarianceThresholdSelector"))
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setVarianceThreshold(value: Double): this.type = set(varianceThreshold, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("3.1.0")
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ @Since("3.1.0")
+ override def fit(dataset: Dataset[_]): VarianceThresholdSelectorModel = {
+ transformSchema(dataset.schema, logging = true)
+
+ val Row(maxs: Vector, mins: Vector, variances: Vector) = dataset
+ .select(Summarizer.metrics("max", "min", "variance").summary(col($(featuresCol)))
+ .as("summary"))
+ .select("summary.max", "summary.min", "summary.variance")
+ .first()
+
+ val numFeatures = maxs.size
+ val indices = Array.tabulate(numFeatures) { i =>
+ (i, if (maxs(i) == mins(i)) 0.0 else variances(i))
Review comment:
I'd like to keep previous comments here :
"# Use peak-to-peak to avoid numeric precision issues for constant features"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-601841635
**[Test build #120113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120113/testReport)** for PR 27954 at commit [`886b2dc`](https://github.com/apache/spark/commit/886b2dca6064118912180d3c511d857963683eeb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27954:
[SPARK-31185][ML] Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-600955160
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120015/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27954: [SPARK-31185][ML]
Implement VarianceThresholdSelector
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27954: [SPARK-31185][ML] Implement VarianceThresholdSelector
URL: https://github.com/apache/spark/pull/27954#issuecomment-602152537
Thank you very much!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org