You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/10 03:56:30 UTC
[GitHub] [spark] zhengruifeng opened a new pull request #28176:
[SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
zhengruifeng opened a new pull request #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176
### What changes were proposed in this pull request?
1, remove newly added method: `def testChiSquare(dataset: Dataset[_], featuresCol: String, labelCol: String): Array[SelectionTestResult]`, because: 1) it is only used in `ChiSqSelector`; 2, since the returned dataframe of `def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame` only contains one row, after collect it back to driver, the results are similar;
2, add method `def test(dataset: DataFrame, featuresCol: String, labelCol: String, flatten: Boolean): DataFrame` to return the flatten results;
### Why are the changes needed?
1, when get returned result dataframe, we may want to filter it like `pValue<0.1`, but current returned dataframe is hard to use;
2, current impl need to collect all test results of all columns back to the driver, this is a bottleneck, if we return the flatten datafame, we no longer to collect them to driver;
### Does this PR introduce any user-facing change?
Yes:
1, `def testChiSquare(dataset: Dataset[_], featuresCol: String, labelCol: String): Array[SelectionTestResult]` removed;
2, the returned dataframe need an action to trigger computation;
### How was this patch tested?
updated testsuites
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612007708
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25776/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r407017158
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -63,40 +55,47 @@ object ChiSquareTest {
*/
@Since("2.2.0")
def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame = {
- val spark = dataset.sparkSession
- import spark.implicits._
-
- SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
- SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val rdd = dataset.select(col(labelCol).cast("double"), col(featuresCol)).as[(Double, Vector)]
- .rdd.map { case (label, features) => OldLabeledPoint(label, OldVectors.fromML(features)) }
- val testResults = OldStatistics.chiSqTest(rdd)
- val pValues = Vectors.dense(testResults.map(_.pValue))
- val degreesOfFreedom = testResults.map(_.degreesOfFreedom)
- val statistics = Vectors.dense(testResults.map(_.statistic))
- spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
+ test(dataset, featuresCol, labelCol, false)
}
/**
* @param dataset DataFrame of categorical labels and categorical features.
* Real-valued features will be treated as categorical for each distinct value.
* @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
* @param labelCol Name of label column in dataset, of any numerical type
- * @return Array containing the SelectionTestResult for every feature against the label.
+ * @param flatten If false, the returned DataFrame contains only a single Row, otherwise, one
Review comment:
> Is the behavior identical to the existing method if false?
Yes, since all methods share the same logic to compute test results, the differences lie in how to represent the results.
> The existing method below seems like it had an OK purpose too, to return the results as objects rather than a DataFrame
I am neutral on keeping it or not. How do you @huaxingao think about it? I can add it back it you want it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611946814
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612350731
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611900856
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25753/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612371836
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121125/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612368305
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25817/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612007702
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r406614726
##########
File path: mllib/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala
##########
@@ -117,15 +117,15 @@ class ChiSquareTestSuite
withClue("ChiSquare should throw an exception when given a continuous-valued label") {
intercept[SparkException] {
val df = spark.createDataFrame(continuousLabel)
- ChiSquareTest.test(df, "features", "label")
+ ChiSquareTest.test(df, "features", "label").count()
Review comment:
need to add an action to trigger the computation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612023523
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r407126004
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -63,40 +55,47 @@ object ChiSquareTest {
*/
@Since("2.2.0")
def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame = {
- val spark = dataset.sparkSession
- import spark.implicits._
-
- SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
- SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val rdd = dataset.select(col(labelCol).cast("double"), col(featuresCol)).as[(Double, Vector)]
- .rdd.map { case (label, features) => OldLabeledPoint(label, OldVectors.fromML(features)) }
- val testResults = OldStatistics.chiSqTest(rdd)
- val pValues = Vectors.dense(testResults.map(_.pValue))
- val degreesOfFreedom = testResults.map(_.degreesOfFreedom)
- val statistics = Vectors.dense(testResults.map(_.statistic))
- spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
+ test(dataset, featuresCol, labelCol, false)
}
/**
* @param dataset DataFrame of categorical labels and categorical features.
* Real-valued features will be treated as categorical for each distinct value.
* @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
* @param labelCol Name of label column in dataset, of any numerical type
- * @return Array containing the SelectionTestResult for every feature against the label.
+ * @param flatten If false, the returned DataFrame contains only a single Row, otherwise, one
Review comment:
I am OK with the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611948929
**[Test build #121074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121074/testReport)** for PR 28176 at commit [`cece841`](https://github.com/apache/spark/commit/cece841c658d06335472fffe6ee832011eb07119).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612572757
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611909551
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121062/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611909551
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121062/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611948929
**[Test build #121074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121074/testReport)** for PR 28176 at commit [`cece841`](https://github.com/apache/spark/commit/cece841c658d06335472fffe6ee832011eb07119).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611927343
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611882567
**[Test build #121055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121055/testReport)** for PR 28176 at commit [`0107dc4`](https://github.com/apache/spark/commit/0107dc47ba432988f686e9019788e76ffc145753).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612572760
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121139/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612380440
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612380440
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611946826
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25765/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612342580
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-613257055
Merged to master, thanks @huaxingao @srowen for reviewing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611944753
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612360149
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25813/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612325436
**[Test build #121119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121119/testReport)** for PR 28176 at commit [`9ec3def`](https://github.com/apache/spark/commit/9ec3def2e25e275ac60f276e8af2eebf7109bec4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r407126758
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
##########
@@ -197,50 +197,54 @@ final class ChiSqSelector @Since("1.6.0") (@Since("1.6.0") override val uid: Str
@Since("2.0.0")
override def fit(dataset: Dataset[_]): ChiSqSelectorModel = {
transformSchema(dataset.schema, logging = true)
- dataset.select(col($(labelCol)).cast(DoubleType), col($(featuresCol))).rdd.map {
- case Row(label: Double, features: Vector) =>
- LabeledPoint(label, features)
+ val spark = dataset.sparkSession
+ import spark.implicits._
+
+ val numFeatures = MetadataUtils.getNumFeatures(dataset, $(featuresCol))
+ val resultDF = ChiSquareTest.test(dataset.toDF, $(featuresCol), $(labelCol), true)
+
+ def getTopIndices(k: Int): Array[Int] = {
+ resultDF.sort("pValue", "featureIndex")
+ .select("featureIndex")
+ .limit(k)
+ .as[Int]
+ .collect()
}
- val testResult = ChiSquareTest.testChiSquare(dataset, getFeaturesCol, getLabelCol)
- .zipWithIndex
- val features = $(selectorType) match {
+ val indices = $(selectorType) match {
case "numTopFeatures" =>
- testResult
- .sortBy { case (res, _) => res.pValue }
- .take(getNumTopFeatures)
+ getTopIndices($(numTopFeatures))
case "percentile" =>
- testResult
- .sortBy { case (res, _) => res.pValue }
- .take((testResult.length * getPercentile).toInt)
+ getTopIndices((numFeatures * getPercentile).toInt)
case "fpr" =>
- testResult
- .filter { case (res, _) => res.pValue < getFpr }
+ resultDF.select("featureIndex")
+ .where(col("pValue").lt($(fpr)))
+ .as[Int].collect()
case "fdr" =>
// This uses the Benjamini-Hochberg procedure.
// https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure
- val tempRes = testResult
- .sortBy { case (res, _) => res.pValue }
- val selected = tempRes
+ val f = $(fdr) / numFeatures
+ val maxIndex = resultDF.sort("pValue", "featureIndex")
+ .select("pValue")
+ .as[Double].rdd
.zipWithIndex
- .filter { case ((res, _), index) =>
- res.pValue <= getFdr * (index + 1) / testResult.length
- }
- if (selected.isEmpty) {
- Array.empty[(SelectionTestResult, Int)]
- } else {
- val maxIndex = selected.map(_._2).max
- tempRes.take(maxIndex + 1)
- }
+ .flatMap { case (pValue, index) =>
+ if (pValue <= f * (index + 1)) {
+ Iterator.single(index.toInt)
+ } else Iterator.empty
+ }.fold(-1)(math.max)
+ if (maxIndex >= 0) {
+ getTopIndices(maxIndex + 1)
+ } else Array.emptyIntArray
case "fwe" =>
- testResult
- .filter { case (res, _) => res.pValue < getFwe / testResult.length }
+ resultDF.select("featureIndex")
+ .where(col("pValue").lt($(fwe) / numFeatures))
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612367173
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25816/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612367173
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25816/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612017261
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25778/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611944753
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612371647
**[Test build #121125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121125/testReport)** for PR 28176 at commit [`67250b7`](https://github.com/apache/spark/commit/67250b75eff70cf1d0839c37b7b24e6479388283).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612350587
**[Test build #121121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121121/testReport)** for PR 28176 at commit [`a6f620f`](https://github.com/apache/spark/commit/a6f620f50b27abc8f5baa765ec0ecca092339ada).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611927346
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25758/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612359889
**[Test build #121125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121125/testReport)** for PR 28176 at commit [`67250b7`](https://github.com/apache/spark/commit/67250b75eff70cf1d0839c37b7b24e6479388283).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612335786
**[Test build #121119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121119/testReport)** for PR 28176 at commit [`9ec3def`](https://github.com/apache/spark/commit/9ec3def2e25e275ac60f276e8af2eebf7109bec4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612350735
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121121/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611926942
**[Test build #121067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121067/testReport)** for PR 28176 at commit [`7a85888`](https://github.com/apache/spark/commit/7a85888c224d0671d0adf433f546f68cec0fbcab).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612567847
**[Test build #121139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121139/testReport)** for PR 28176 at commit [`6db77f2`](https://github.com/apache/spark/commit/6db77f292d0b47b4b8b2d282ec8547fa57d507fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612368131
**[Test build #121128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121128/testReport)** for PR 28176 at commit [`0f2413d`](https://github.com/apache/spark/commit/0f2413d3401a3229eba1a64ff01ae19518ac9980).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611970377
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611946826
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25765/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612380256
**[Test build #121128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121128/testReport)** for PR 28176 at commit [`0f2413d`](https://github.com/apache/spark/commit/0f2413d3401a3229eba1a64ff01ae19518ac9980).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612007327
**[Test build #121086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121086/testReport)** for PR 28176 at commit [`16e4ded`](https://github.com/apache/spark/commit/16e4ded0ed4456851624520fd5d071489b20e607).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612567942
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611926942
**[Test build #121067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121067/testReport)** for PR 28176 at commit [`7a85888`](https://github.com/apache/spark/commit/7a85888c224d0671d0adf433f546f68cec0fbcab).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612360145
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612007327
**[Test build #121086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121086/testReport)** for PR 28176 at commit [`16e4ded`](https://github.com/apache/spark/commit/16e4ded0ed4456851624520fd5d071489b20e607).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612342398
**[Test build #121121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121121/testReport)** for PR 28176 at commit [`a6f620f`](https://github.com/apache/spark/commit/a6f620f50b27abc8f5baa765ec0ecca092339ada).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612360145
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611882920
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612350731
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612342580
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612360149
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25813/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611909545
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611900856
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25753/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612007708
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25776/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611882927
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121055/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612335935
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612380445
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121128/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612567942
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611944478
**[Test build #121067 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121067/testReport)** for PR 28176 at commit [`7a85888`](https://github.com/apache/spark/commit/7a85888c224d0671d0adf433f546f68cec0fbcab).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r407126920
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -63,40 +55,50 @@ object ChiSquareTest {
*/
@Since("2.2.0")
def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame = {
- val spark = dataset.sparkSession
- import spark.implicits._
-
- SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
- SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val rdd = dataset.select(col(labelCol).cast("double"), col(featuresCol)).as[(Double, Vector)]
- .rdd.map { case (label, features) => OldLabeledPoint(label, OldVectors.fromML(features)) }
- val testResults = OldStatistics.chiSqTest(rdd)
- val pValues = Vectors.dense(testResults.map(_.pValue))
- val degreesOfFreedom = testResults.map(_.degreesOfFreedom)
- val statistics = Vectors.dense(testResults.map(_.statistic))
- spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
+ test(dataset, featuresCol, labelCol, false)
}
/**
* @param dataset DataFrame of categorical labels and categorical features.
* Real-valued features will be treated as categorical for each distinct value.
* @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
* @param labelCol Name of label column in dataset, of any numerical type
- * @return Array containing the SelectionTestResult for every feature against the label.
+ * @param flatten If false, the returned DataFrame contains only a single Row, otherwise, one
+ * row per feature.
*/
@Since("3.1.0")
- def testChiSquare(
- dataset: Dataset[_],
+ def test(
+ dataset: DataFrame,
featuresCol: String,
- labelCol: String): Array[SelectionTestResult] = {
-
+ labelCol: String,
+ flatten: Boolean): DataFrame = {
SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val input = dataset.select(col(labelCol).cast(DoubleType), col(featuresCol)).rdd
- .map { case Row(label: Double, features: Vector) =>
- OldLabeledPoint(label, OldVectors.fromML(features))
- }
- val chiTestResult = OldStatistics.chiSqTest(input)
- chiTestResult.map(r => new ChiSqTestResult(r.pValue, r.degreesOfFreedom, r.statistic))
+
+ val spark = dataset.sparkSession
+ import spark.implicits._
+
+ val data = dataset.select(col(labelCol).cast("double"), col(featuresCol)).rdd
+ .map { case Row(label: Double, vec: Vector) => (label, OldVectors.fromML(vec)) }
+
+ val resultDF = OldChiSqTest.computeChiSquared(data)
+ .map { case (col, pValue, degreesOfFreedom, statistic, _) =>
+ (col, pValue, degreesOfFreedom, statistic)
+ }.toDF("featureIndex", "pValue", "degreesOfFreedom", "statistic")
+
+ if (flatten) {
+ resultDF
+ } else {
+ resultDF.groupBy()
Review comment:
nit: do we need the ```groupBy()``` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612007702
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612335942
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121119/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612017256
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612325740
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25807/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612342583
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25809/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612380445
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121128/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611944757
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121067/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612368301
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612368301
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611867286
**[Test build #121055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121055/testReport)** for PR 28176 at commit [`0107dc4`](https://github.com/apache/spark/commit/0107dc47ba432988f686e9019788e76ffc145753).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611900485
**[Test build #121062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121062/testReport)** for PR 28176 at commit [`3ae2551`](https://github.com/apache/spark/commit/3ae255130144bedf620a85863ce19369283eacd4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612325734
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612367167
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612023224
**[Test build #121086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121086/testReport)** for PR 28176 at commit [`16e4ded`](https://github.com/apache/spark/commit/16e4ded0ed4456851624520fd5d071489b20e607).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612039406
**[Test build #121088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121088/testReport)** for PR 28176 at commit [`02bc72b`](https://github.com/apache/spark/commit/02bc72b05329780709e1758a993dfd357aceb693).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611970377
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612019112
**[Test build #121088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121088/testReport)** for PR 28176 at commit [`02bc72b`](https://github.com/apache/spark/commit/02bc72b05329780709e1758a993dfd357aceb693).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612572757
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612567947
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25827/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612342398
**[Test build #121121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121121/testReport)** for PR 28176 at commit [`a6f620f`](https://github.com/apache/spark/commit/a6f620f50b27abc8f5baa765ec0ecca092339ada).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612371827
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611882927
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121055/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612371827
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611867467
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25746/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611892576
friendly ping @srowen @huaxingao
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611867466
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612335935
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612350735
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121121/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611927346
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25758/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612017261
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25778/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612572760
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121139/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611944757
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121067/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612023528
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121086/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r407149741
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -63,40 +55,50 @@ object ChiSquareTest {
*/
@Since("2.2.0")
def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame = {
- val spark = dataset.sparkSession
- import spark.implicits._
-
- SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
- SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val rdd = dataset.select(col(labelCol).cast("double"), col(featuresCol)).as[(Double, Vector)]
- .rdd.map { case (label, features) => OldLabeledPoint(label, OldVectors.fromML(features)) }
- val testResults = OldStatistics.chiSqTest(rdd)
- val pValues = Vectors.dense(testResults.map(_.pValue))
- val degreesOfFreedom = testResults.map(_.degreesOfFreedom)
- val statistics = Vectors.dense(testResults.map(_.statistic))
- spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
+ test(dataset, featuresCol, labelCol, false)
}
/**
* @param dataset DataFrame of categorical labels and categorical features.
* Real-valued features will be treated as categorical for each distinct value.
* @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
* @param labelCol Name of label column in dataset, of any numerical type
- * @return Array containing the SelectionTestResult for every feature against the label.
+ * @param flatten If false, the returned DataFrame contains only a single Row, otherwise, one
+ * row per feature.
*/
@Since("3.1.0")
- def testChiSquare(
- dataset: Dataset[_],
+ def test(
+ dataset: DataFrame,
featuresCol: String,
- labelCol: String): Array[SelectionTestResult] = {
-
+ labelCol: String,
+ flatten: Boolean): DataFrame = {
SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val input = dataset.select(col(labelCol).cast(DoubleType), col(featuresCol)).rdd
- .map { case Row(label: Double, features: Vector) =>
- OldLabeledPoint(label, OldVectors.fromML(features))
- }
- val chiTestResult = OldStatistics.chiSqTest(input)
- chiTestResult.map(r => new ChiSqTestResult(r.pValue, r.degreesOfFreedom, r.statistic))
+
+ val spark = dataset.sparkSession
+ import spark.implicits._
+
+ val data = dataset.select(col(labelCol).cast("double"), col(featuresCol)).rdd
+ .map { case Row(label: Double, vec: Vector) => (label, OldVectors.fromML(vec)) }
+
+ val resultDF = OldChiSqTest.computeChiSquared(data)
+ .map { case (col, pValue, degreesOfFreedom, statistic, _) =>
+ (col, pValue, degreesOfFreedom, statistic)
+ }.toDF("featureIndex", "pValue", "degreesOfFreedom", "statistic")
+
+ if (flatten) {
+ resultDF
+ } else {
+ resultDF.groupBy()
Review comment:
yes, this `groupBy` is needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611900853
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612368305
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25817/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612019112
**[Test build #121088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121088/testReport)** for PR 28176 at commit [`02bc72b`](https://github.com/apache/spark/commit/02bc72b05329780709e1758a993dfd357aceb693).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r406614269
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -63,40 +55,47 @@ object ChiSquareTest {
*/
@Since("2.2.0")
def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame = {
- val spark = dataset.sparkSession
- import spark.implicits._
-
- SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
- SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val rdd = dataset.select(col(labelCol).cast("double"), col(featuresCol)).as[(Double, Vector)]
- .rdd.map { case (label, features) => OldLabeledPoint(label, OldVectors.fromML(features)) }
- val testResults = OldStatistics.chiSqTest(rdd)
- val pValues = Vectors.dense(testResults.map(_.pValue))
- val degreesOfFreedom = testResults.map(_.degreesOfFreedom)
- val statistics = Vectors.dense(testResults.map(_.statistic))
- spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
+ test(dataset, featuresCol, labelCol, false)
}
/**
* @param dataset DataFrame of categorical labels and categorical features.
* Real-valued features will be treated as categorical for each distinct value.
* @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
* @param labelCol Name of label column in dataset, of any numerical type
- * @return Array containing the SelectionTestResult for every feature against the label.
+ * @param flatten If false, the returned DataFrame contains only a single Row, otherwise, one
+ * row per feature.
*/
@Since("3.1.0")
- def testChiSquare(
- dataset: Dataset[_],
+ def test(
+ dataset: DataFrame,
featuresCol: String,
- labelCol: String): Array[SelectionTestResult] = {
-
+ labelCol: String,
+ flatten: Boolean): DataFrame = {
SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val input = dataset.select(col(labelCol).cast(DoubleType), col(featuresCol)).rdd
- .map { case Row(label: Double, features: Vector) =>
- OldLabeledPoint(label, OldVectors.fromML(features))
- }
- val chiTestResult = OldStatistics.chiSqTest(input)
- chiTestResult.map(r => new ChiSqTestResult(r.pValue, r.degreesOfFreedom, r.statistic))
+
+ val spark = dataset.sparkSession
+ import spark.implicits._
+
+ val data = dataset.select(col(labelCol).cast("double"), col(featuresCol)).rdd
+ .map { case Row(label: Double, vec: Vector) => (label, OldVectors.fromML(vec)) }
+ val resultRDD = OldChiSqTest.computeChiSquared(data)
+
+ if (flatten) {
+ resultRDD.map { case (col, (pValue, degreesOfFreedom, statistic, _)) =>
+ (col, pValue, degreesOfFreedom, statistic)
+ }.toDF("featureIndex", "pValue", "degreesOfFreedom", "statistic")
+ } else {
+ resultRDD.map { case (col, (pValue, degreesOfFreedom, statistic, _)) =>
+ (0, (col, pValue, degreesOfFreedom, statistic))
+ }.groupByKey().map { case (_, seq) =>
Review comment:
collect results in some executor instead of the driver
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r406614762
##########
File path: mllib/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala
##########
@@ -117,15 +117,15 @@ class ChiSquareTestSuite
withClue("ChiSquare should throw an exception when given a continuous-valued label") {
intercept[SparkException] {
val df = spark.createDataFrame(continuousLabel)
- ChiSquareTest.test(df, "features", "label")
+ ChiSquareTest.test(df, "features", "label").count()
}
}
val continuousFeature = Seq.fill(tooManyCategories)(
LabeledPoint(random.nextInt(2), Vectors.dense(random.nextDouble())))
withClue("ChiSquare should throw an exception when given continuous-valued features") {
intercept[SparkException] {
val df = spark.createDataFrame(continuousFeature)
- ChiSquareTest.test(df, "features", "label")
+ ChiSquareTest.test(df, "features", "label").count()
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611882920
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612567847
**[Test build #121139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121139/testReport)** for PR 28176 at commit [`6db77f2`](https://github.com/apache/spark/commit/6db77f292d0b47b4b8b2d282ec8547fa57d507fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r407126724
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
##########
@@ -197,50 +197,54 @@ final class ChiSqSelector @Since("1.6.0") (@Since("1.6.0") override val uid: Str
@Since("2.0.0")
override def fit(dataset: Dataset[_]): ChiSqSelectorModel = {
transformSchema(dataset.schema, logging = true)
- dataset.select(col($(labelCol)).cast(DoubleType), col($(featuresCol))).rdd.map {
- case Row(label: Double, features: Vector) =>
- LabeledPoint(label, features)
+ val spark = dataset.sparkSession
+ import spark.implicits._
+
+ val numFeatures = MetadataUtils.getNumFeatures(dataset, $(featuresCol))
+ val resultDF = ChiSquareTest.test(dataset.toDF, $(featuresCol), $(labelCol), true)
+
+ def getTopIndices(k: Int): Array[Int] = {
+ resultDF.sort("pValue", "featureIndex")
+ .select("featureIndex")
+ .limit(k)
+ .as[Int]
+ .collect()
}
- val testResult = ChiSquareTest.testChiSquare(dataset, getFeaturesCol, getLabelCol)
- .zipWithIndex
- val features = $(selectorType) match {
+ val indices = $(selectorType) match {
case "numTopFeatures" =>
- testResult
- .sortBy { case (res, _) => res.pValue }
- .take(getNumTopFeatures)
+ getTopIndices($(numTopFeatures))
case "percentile" =>
- testResult
- .sortBy { case (res, _) => res.pValue }
- .take((testResult.length * getPercentile).toInt)
+ getTopIndices((numFeatures * getPercentile).toInt)
case "fpr" =>
- testResult
- .filter { case (res, _) => res.pValue < getFpr }
+ resultDF.select("featureIndex")
+ .where(col("pValue").lt($(fpr)))
Review comment:
super nit: use ```<``` instead of ```lt```?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612023528
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121086/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612325436
**[Test build #121119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121119/testReport)** for PR 28176 at commit [`9ec3def`](https://github.com/apache/spark/commit/9ec3def2e25e275ac60f276e8af2eebf7109bec4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612039663
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121088/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612335942
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121119/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612572679
**[Test build #121139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121139/testReport)** for PR 28176 at commit [`6db77f2`](https://github.com/apache/spark/commit/6db77f292d0b47b4b8b2d282ec8547fa57d507fd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612039663
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121088/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612359889
**[Test build #121125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121125/testReport)** for PR 28176 at commit [`67250b7`](https://github.com/apache/spark/commit/67250b75eff70cf1d0839c37b7b24e6479388283).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612023523
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612368131
**[Test build #121128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121128/testReport)** for PR 28176 at commit [`0f2413d`](https://github.com/apache/spark/commit/0f2413d3401a3229eba1a64ff01ae19518ac9980).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612325734
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612039656
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612017256
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612567947
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25827/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611867467
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25746/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611946814
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611909545
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611927343
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612342583
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25809/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611875587
bin/spark-shell --driver-memory=2G
testcodes:
```
import org.apache.spark.ml.feature.ChiSqSelector
import org.apache.spark.ml.stat._
val df = spark.read.format("libsvm").load("/data1/Datasets/webspam/webspam_wc_normalized_trigram.svm.10k")
val chi = ChiSquareTest.test(df, "features", "label")
// val chi = ChiSquareTest.test(df, "features", "label", true) // added in this PR
chi.show
val selector = new ChiSqSelector().setNumTopFeatures(1000).setLabelCol("label").setFeaturesCol("features")
val model = selector.fit(df)
```
Existing `ChiSquareTest.test` and `selector.fit` will crash Spark-Shell due to OOM; while new methods in this PR work fine:
Existing `ChiSquareTest.test`
```
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2094)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2043)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2042)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2042)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1020)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1020)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1020)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2274)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2223)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2212)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:822)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2108)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2129)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2148)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2173)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
at org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredSparseFeatures(ChiSqTest.scala:148)
at org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredFeatures(ChiSqTest.scala:88)
at org.apache.spark.mllib.stat.Statistics$.chiSqTest(Statistics.scala:192)
at org.apache.spark.ml.stat.ChiSquareTest$.test(ChiSquareTest.scala:73)
... 49 elided
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
```
new `ChiSquareTest.test(df, "features", "label", true)`
```
scala> val chi = ChiSquareTest.test(df, "features", "label", true)
chi: org.apache.spark.sql.DataFrame = [featureIndex: int, pValue: double ... 2 more fields]
scala> chi.show
20/04/10 11:28:51 WARN Executor: Managed memory leak detected; size = 843044942 bytes, TID = 49
+------------+-------------------+----------------+------------------+
|featureIndex| pValue|degreesOfFreedom| statistic|
+------------+-------------------+----------------+------------------+
| 3184020| 1.0| 0| 0.0|
| 6697512|0.21072480020843432| 2|3.1144045224283925|
| 3387408| 1.0| 0| 0.0|
| 5907828| 1.0| 0| 0.0|
| 6582516| 1.0| 0| 0.0|
| 3490824| 1.0| 0| 0.0|
| 5916408| 1.0| 0| 0.0|
| 2292732| 1.0| 0| 0.0|
| 7447896| 1.0| 0| 0.0|
| 2157804| 1.0| 0| 0.0|
| 1732608| 1.0| 0| 0.0|
| 3427284| 1.0| 0| 0.0|
| 5284836| 1.0| 0| 0.0|
| 3812544| 1.0| 0| 0.0|
| 2396784| 1.0| 0| 0.0|
| 5761524| 1.0| 0| 0.0|
| 3138636| 1.0| 0| 0.0|
| 6971832| 1.0| 0| 0.0|
| 8164968| 1.0| 0| 0.0|
| 2321808| 1.0| 0| 0.0|
+------------+-------------------+----------------+------------------+
only showing top 20 rows
```
new `selector.fit(df)`
```
scala> val model = selector.fit(df)
model: org.apache.spark.ml.feature.ChiSqSelectorModel = ChiSqSelectorModel: uid=chiSqSelector_3b712e166c6f, numSelectedFeatures=1000
scala> model.selectedFeatures
res0: Array[Int] = Array(592236, 592238, 605261, 614717, 616224, 617577, 619040, 663584, 670767, 670792, 670797, 670804, 670817, 670820, 670828, 670832, 2105411, 2105903, 2105921, 2105972, 2106179, 2110000, 2111536, 2112559, 2112621, 2112622, 2112627, 2113858, 2114061, 2115668, 2115950, 2117221, 2122083, 2122092, 2122094, 2122340, 2122351, 2122604, 2122607, 2123375, 2123877, 2123890, 2124132, 2124142, 2124147, 2124897, 2125157, 2125409, 2125678, 2125938, 2126437, 2126697, 2126708, 2126949, 2126953, 2126969, 2127457, 2127721, 2171454, 2236513, 2236515, 2236520, 2236531, 2236532, 2236534, 2236535, 2236960, 2236990, 2237251, 2237253, 2237254, 2239522, 2239541, 2240353, 2240827, 2241058, 2241841, 2242104, 2244106, 2244156, 2244161, 2244162, 2244168, 2244169, 224417...
```
Since `ChiSquareTest` and `ChiSqSelector` are for feature selection, so I think it is usual that the input has high dimensionality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612367167
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611970384
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121074/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612325740
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25807/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611970384
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121074/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611900853
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611900485
**[Test build #121062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121062/testReport)** for PR 28176 at commit [`3ae2551`](https://github.com/apache/spark/commit/3ae255130144bedf620a85863ce19369283eacd4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611867286
**[Test build #121055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121055/testReport)** for PR 28176 at commit [`0107dc4`](https://github.com/apache/spark/commit/0107dc47ba432988f686e9019788e76ffc145753).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612039656
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-612371836
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121125/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176:
[SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611867466
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611909488
**[Test build #121062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121062/testReport)** for PR 28176 at commit [`3ae2551`](https://github.com/apache/spark/commit/3ae255130144bedf620a85863ce19369283eacd4).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611970058
**[Test build #121074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121074/testReport)** for PR 28176 at commit [`cece841`](https://github.com/apache/spark/commit/cece841c658d06335472fffe6ee832011eb07119).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #28176:
[SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r406774357
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -63,40 +55,47 @@ object ChiSquareTest {
*/
@Since("2.2.0")
def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame = {
- val spark = dataset.sparkSession
- import spark.implicits._
-
- SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
- SchemaUtils.checkNumericType(dataset.schema, labelCol)
- val rdd = dataset.select(col(labelCol).cast("double"), col(featuresCol)).as[(Double, Vector)]
- .rdd.map { case (label, features) => OldLabeledPoint(label, OldVectors.fromML(features)) }
- val testResults = OldStatistics.chiSqTest(rdd)
- val pValues = Vectors.dense(testResults.map(_.pValue))
- val degreesOfFreedom = testResults.map(_.degreesOfFreedom)
- val statistics = Vectors.dense(testResults.map(_.statistic))
- spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
+ test(dataset, featuresCol, labelCol, false)
}
/**
* @param dataset DataFrame of categorical labels and categorical features.
* Real-valued features will be treated as categorical for each distinct value.
* @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
* @param labelCol Name of label column in dataset, of any numerical type
- * @return Array containing the SelectionTestResult for every feature against the label.
+ * @param flatten If false, the returned DataFrame contains only a single Row, otherwise, one
Review comment:
Is the behavior identical to the existing method if false?
Hm OK this is a little odd API-wise but I understand you're trying to keep compatibility while adding a better representation too. I'm neutral on it.
The existing method below seems like it had an OK purpose too, to return the results as objects rather than a DataFrame, but I also don't feel strongly about retaining it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org