You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/07 00:47:26 UTC
[GitHub] [spark] huaxingao opened a new pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
huaxingao opened a new pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841
### What changes were proposed in this pull request?
```ChiSqSelector ``` depends on ```mllib.ChiSqSelectorModel``` to do the selection logic. Will remove the dependency in this PR.
### Why are the changes needed?
This PR is an intermediate PR. Removing ```ChiSqSelector``` dependency on ```mllib.ChiSqSelectorModel```. Next subtask will extract the common code between ```ChiSqSelector``` and ```FValueSelector``` and put in an abstract ```Selector```.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
New and existing tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596935633
**[Test build #119601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119601/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r389673912
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
##########
@@ -270,10 +293,10 @@ final class ChiSqSelectorModel private[ml] (
val func = { vector: Vector =>
vector match {
case SparseVector(_, indices, values) =>
- val (newIndices, newValues) = chiSqSelector.compressSparse(indices, values)
+ val (newIndices, newValues) = compressSparse(indices, values)
Vectors.sparse(newSize, newIndices, newValues)
case DenseVector(values) =>
- Vectors.dense(chiSqSelector.compressDense(values))
+ Vectors.dense(selectedFeatures.map(i => values(i)))
Review comment:
does it work to just `.map(values)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597079415
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24348/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596941747
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119606/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596941730
**[Test build #119606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119606/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596121225
**[Test build #119520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119520/testReport)** for PR 27841 at commit [`bf4a819`](https://github.com/apache/spark/commit/bf4a81992a74da5ea107388b20292ed820e55e32).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596022914
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596941747
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119606/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596133542
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596022918
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24225/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596940473
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597699390
If you're OK with this in its current form I will merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596041640
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119497/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597809773
Thanks! @srowen @zhengruifeng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596941741
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596617162
Fixed. Thank you very much! @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596121375
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597167960
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24353/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596712160
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119574/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596901578
**[Test build #119601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119601/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596133547
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119520/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596149486
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597167195
**[Test build #119624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119624/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r390077499
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
##########
@@ -245,14 +266,15 @@ object ChiSqSelector extends DefaultParamsReadable[ChiSqSelector] {
@Since("1.6.0")
final class ChiSqSelectorModel private[ml] (
@Since("1.6.0") override val uid: String,
- private val chiSqSelector: feature.ChiSqSelectorModel)
+ @Since("3.1.0")val selectedFeatures: Array[Int])
Review comment:
nit: missing space before `val`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597167195
**[Test build #119624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119624/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596938482
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596121225
**[Test build #119520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119520/testReport)** for PR 27841 at commit [`bf4a819`](https://github.com/apache/spark/commit/bf4a81992a74da5ea107388b20292ed820e55e32).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596041638
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r390092960
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -75,4 +77,31 @@ object ChiSquareTest {
val statistics: Vector = Vectors.dense(testResults.map(_.statistic))
spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
}
+
+ /**
+ * @param dataset DataFrame of categorical labels and categorical features.
+ * Real-valued features will be treated as categorical for each distinct value.
+ * @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return Array containing the SelectionTestResult for every feature against the label.
+ */
+ @Since("3.1.0")
+ def testChiSquare(
+ dataset: Dataset[_],
+ featuresCol: String,
+ labelCol: String): Array[SelectionTestResult] = {
Review comment:
I will need to have an abstract method in abstract class ```Selector``` in next subtask
```
def getSelectionTestResult(dataset: Dataset[_]): Array[SelectionTestResult]
```
so I prefer to return ```Array[SelectionTestResult]``` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597254346
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597078796
**[Test build #119619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119619/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597164874
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597254352
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119624/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596137171
**[Test build #119522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119522/testReport)** for PR 27841 at commit [`bf4a819`](https://github.com/apache/spark/commit/bf4a81992a74da5ea107388b20292ed820e55e32).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597145376
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119619/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r390078381
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -75,4 +77,31 @@ object ChiSquareTest {
val statistics: Vector = Vectors.dense(testResults.map(_.statistic))
spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
}
+
+ /**
+ * @param dataset DataFrame of categorical labels and categorical features.
+ * Real-valued features will be treated as categorical for each distinct value.
+ * @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return Array containing the SelectionTestResult for every feature against the label.
+ */
+ @Since("3.1.0")
+ def testChiSquare(
+ dataset: Dataset[_],
+ featuresCol: String,
+ labelCol: String): Array[SelectionTestResult] = {
Review comment:
What about returning `Array[ChiSquareResult]`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597145376
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119619/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596901801
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r390092933
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -75,4 +77,31 @@ object ChiSquareTest {
val statistics: Vector = Vectors.dense(testResults.map(_.statistic))
spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
}
+
+ /**
+ * @param dataset DataFrame of categorical labels and categorical features.
+ * Real-valued features will be treated as categorical for each distinct value.
+ * @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return Array containing the SelectionTestResult for every feature against the label.
+ */
+ @Since("3.1.0")
Review comment:
Yes.
I may need to refactor this method and move it out of this class in next subtask.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596149487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119522/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596022918
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24225/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596901801
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597079415
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24348/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596901809
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24331/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596136640
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596939996
**[Test build #119606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119606/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596041638
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596617168
**[Test build #119574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119574/testReport)** for PR 27841 at commit [`4eaf24c`](https://github.com/apache/spark/commit/4eaf24cd042dfe2d43ef9dd4ddbddbb140835477).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596940479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24335/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596120093
@srowen Thank you very much for your review!
I am trying to remove the dependency on mllib.ChiSqSelectorModel. After the change, ChiSqSelector.fit, ChiSqSelectorModel constructor and implementation are almost the same as the FValueSelector implementation, then I will extract all the common code between ```ChiSqSelector``` and ```FValueSelector``` to put in an abstract ```Selector```.
So basically here is the final goal:
```
abstract class Selector[T <: SelectorModel[T]]
it contains all the params, fit method + 2 abstract methods:
// Compute the statistics information (pValue + chisq statistics/fvalue)
protected[this] def getSelectionTestResult(dataset: Dataset[_]): Array[SelectionTestResult]
// Create SelectorModel (could be ChiSqModel or FValueModel or ANOVAModel)
protected[this] def createSelectorModel(
uid: String,
statistics: Array[Double]): T
```
Then I will have three concrete Selector classes:
```
1.
// categorical features and categorical labels.
class ChiSqSelector extends Selector[ChiSqSelectorModel]
implement getSelectionTestResult to return pValue and chisq stats
implement createSelectorModel to create a ChiSqSelectorModel
2.
// continuous features and continuous labels
class FValueSelector extends Selector[FValueSelectorModel]
implement getSelectionTestResult to return pValue and f regression value
implement FValueSelector to create a FValueSelectorModel
3.
// continuous features and categorical labels
class ANOVASelector extends Selector[ANOVASelectorModel]
implement getSelectionTestResult to return pValue and ANOVA fvalue
implement ANOVASelector to create a ANOVASelectorModel
```
The initial PR is https://github.com/apache/spark/pull/27527. I broke it to several small PRs so it will be easier to review:
1. add stat.FValueRegressionTest (https://github.com/apache/spark/pull/27623)
2. add feature.FValueSelector (https://github.com/apache/spark/pull/27679)
3. this PR to remove ChiSqSelector dependency on mllib.ChiSqSelectorModel.
4. extract all the common code to abstract Selector. Make FValueSelector and ChiSqSelector implement abstract Selector.
5. Add ANOVASelector (the implementation of ANOVA is not in https://github.com/apache/spark/pull/27527)
Maybe I should combine step 3 and 4 in one PR?
For the binary test files changes: I initially planned to exposed pValue and chisq statistics in ChisqSelectorModel so the constructor is like this:
```
final class ChiSqSelectorModel private[ml] (
@Since("1.6.0") override val uid: String,
override val selectedFeatures: Array[Int],
@Since("3.1.0") override val pValues: Array[Double],
@Since("3.1.0")override val statistic: Array[Double])
```
Then when I load the model, I will need to do this:
```
val model = if (majorVersion < 3 || (majorVersion == 3 && minorVersion < 1)) {
// model prior to 3.1.0
val data = df.select("selectedFeatures").head()
...
} else {
val data = df.select("selectedFeatures", "pValue", "statistics").head()
...
}
```
So I have the binary test files to test the prior 3.1.0 model. But then I wasn't so sure if I need to expose pValue and chisq statistics, and I decided to rethink this later in the final step when I do the abstract Selector. I changed the constructor and reader code but forgot to remove the binary test files. I will clean up now. Sorry my mistake confused you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596137242
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24251/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597145359
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r389697685
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -75,4 +77,34 @@ object ChiSquareTest {
val statistics: Vector = Vectors.dense(testResults.map(_.statistic))
spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
}
+
+ /**
+ * @param dataset DataFrame of categorical labels and categorical features.
+ * Real-valued features will be treated as categorical for each distinct value.
+ * @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return Array containing the SelectionTestResult for every feature against the label.
+ */
+ @Since("3.1.0")
+ def testChiSquare(dataset: Dataset[_], featuresCol: String, labelCol: String):
+ Array[SelectionTestResult] = {
Review comment:
Nit: break this a little differently? indent more here or wrap the arguments and indent
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597807518
Merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596935907
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596618396
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596121376
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24249/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596023910
**[Test build #119497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119497/testReport)** for PR 27841 at commit [`3341122`](https://github.com/apache/spark/commit/3341122156c3c383282b869319e4f1f01ff2ea5c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596149487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119522/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596940473
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597706560
Please merge. Thank you very much! @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596137241
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596901578
**[Test build #119601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119601/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596041487
**[Test build #119497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119497/testReport)** for PR 27841 at commit [`3341122`](https://github.com/apache/spark/commit/3341122156c3c383282b869319e4f1f01ff2ea5c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597079397
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596041640
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119497/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596939996
**[Test build #119606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119606/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596618431
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24305/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596935915
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119601/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596125450
Up to you how you want to proceed.
OK so in the end this should increase code sharing; it's not separating the implementations by copying.
p values do sound useful in the model; up to you. Yes it just needs to be backwards-compatible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596901809
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24331/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596941741
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596618396
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596023910
**[Test build #119497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119497/testReport)** for PR 27841 at commit [`3341122`](https://github.com/apache/spark/commit/3341122156c3c383282b869319e4f1f01ff2ea5c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597078302
Jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596121376
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24249/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596712149
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r389698830
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -75,4 +77,34 @@ object ChiSquareTest {
val statistics: Vector = Vectors.dense(testResults.map(_.statistic))
spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
}
+
+ /**
+ * @param dataset DataFrame of categorical labels and categorical features.
+ * Real-valued features will be treated as categorical for each distinct value.
+ * @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return Array containing the SelectionTestResult for every feature against the label.
+ */
+ @Since("3.1.0")
+ def testChiSquare(dataset: Dataset[_], featuresCol: String, labelCol: String):
+ Array[SelectionTestResult] = {
+
+ val spark = dataset.sparkSession
+
+ SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
+ SchemaUtils.checkNumericType(dataset.schema, labelCol)
+ val input: RDD[OldLabeledPoint] =
+ dataset.select(col(labelCol).cast(DoubleType), col(featuresCol)).rdd
+ .map {
+ case Row(label: Double, features: Vector) =>
+ OldLabeledPoint(label, OldVectors.fromML(features))
+ }
+ val chiTestResult = OldStatistics.chiSqTest(input)
+ var chiTestResultArray = new Array[SelectionTestResult](chiTestResult.length)
Review comment:
Another nit but I think the remainder of the function could be:
```
chiTestResult.map(r => new ChiSqTestResult(r.pValue, r.degreesOfFreedom, r.statistic)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596617168
**[Test build #119574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119574/testReport)** for PR 27841 at commit [`4eaf24c`](https://github.com/apache/spark/commit/4eaf24cd042dfe2d43ef9dd4ddbddbb140835477).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597078796
**[Test build #119619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119619/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596137241
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597167960
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24353/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597145359
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596102294
General questions: does this mean not being able to share as much code? or how do we avoid parallel implementations if one doesn't depend on the other -- they can both share common code elsewhere?
Also why do some of the binary test files change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596935907
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597079397
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596712149
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597144684
**[Test build #119619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119619/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596137242
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24251/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597167953
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r389673662
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
##########
@@ -245,14 +266,16 @@ object ChiSqSelector extends DefaultParamsReadable[ChiSqSelector] {
@Since("1.6.0")
final class ChiSqSelectorModel private[ml] (
@Since("1.6.0") override val uid: String,
- private val chiSqSelector: feature.ChiSqSelectorModel)
+ @Since("3.1.0")val selectedFeatures: Array[Int])
extends Model[ChiSqSelectorModel] with ChiSqSelectorParams with MLWritable {
import ChiSqSelectorModel._
- /** list of indices to select (filter). */
- @Since("1.6.0")
- val selectedFeatures: Array[Int] = chiSqSelector.selectedFeatures
+ var prev = -1
Review comment:
Doesn't matter, but you can write `require(selectedFeatures.sliding(2).forall(l => l(0) < l(1)), ...)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596133542
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596710907
**[Test build #119574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119574/testReport)** for PR 27841 at commit [`4eaf24c`](https://github.com/apache/spark/commit/4eaf24cd042dfe2d43ef9dd4ddbddbb140835477).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596149363
**[Test build #119522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119522/testReport)** for PR 27841 at commit [`bf4a819`](https://github.com/apache/spark/commit/bf4a81992a74da5ea107388b20292ed820e55e32).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#discussion_r390077290
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##########
@@ -75,4 +77,31 @@ object ChiSquareTest {
val statistics: Vector = Vectors.dense(testResults.map(_.statistic))
spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom, statistics)))
}
+
+ /**
+ * @param dataset DataFrame of categorical labels and categorical features.
+ * Real-valued features will be treated as categorical for each distinct value.
+ * @param featuresCol Name of features column in dataset, of type `Vector` (`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return Array containing the SelectionTestResult for every feature against the label.
+ */
+ @Since("3.1.0")
Review comment:
That difference between this method and above `def test(dataset: DataFrame, featuresCol: String, labelCol: String): DataFrame` is only that the result are parallelized to a dataframe?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596618431
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24305/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596940479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24335/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597253274
**[Test build #119624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119624/testReport)** for PR 27841 at commit [`20629ad`](https://github.com/apache/spark/commit/20629ad0ce43a4af9cf94e3993d778d6a8c63ebd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597254346
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596149486
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596022914
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596712160
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119574/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596935915
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119601/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597254352
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119624/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596121375
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27841: [SPARK-31077][ML] Remove
ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596133465
**[Test build #119520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119520/testReport)** for PR 27841 at commit [`bf4a819`](https://github.com/apache/spark/commit/bf4a81992a74da5ea107388b20292ed820e55e32).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-597167953
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27841:
[SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596137171
**[Test build #119522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119522/testReport)** for PR 27841 at commit [`bf4a819`](https://github.com/apache/spark/commit/bf4a81992a74da5ea107388b20292ed820e55e32).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27841: [SPARK-31077][ML]
Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27841: [SPARK-31077][ML] Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel
URL: https://github.com/apache/spark/pull/27841#issuecomment-596133547
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119520/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org