You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/10 04:14:44 UTC
[GitHub] [spark] zhengruifeng opened a new pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
zhengruifeng opened a new pull request #29998:
URL: https://github.com/apache/spark/pull/29998
### What changes were proposed in this pull request?
1, use `blockSizeInMB` instead of `blockSize`(#rows) to control the stacking of vectors;
2, infer an appropriate `blockSizeInMB` if set 0, based on the performance test;
### Why are the changes needed?
the performance gain is mainly related to the nnz of block.
### Does this PR introduce _any_ user-facing change?
yes, param `blockSize` -> `blockSizeInMB` in master
### How was this patch tested?
existing testsuites and performance test (result attached in [ticket](https://issues.apache.org/jira/browse/SPARK-32907))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706488179
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129613/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706488176
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706496666
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706488159
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34216/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706493312
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34218/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706495796
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706514246
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34224/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on a change in pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on a change in pull request #29998:
URL: https://github.com/apache/spark/pull/29998#discussion_r503151315
##########
File path: python/pyspark/resource/profile.pyi
##########
@@ -22,7 +22,7 @@ from pyspark.resource.requests import ( # noqa: F401
TaskResourceRequest as TaskResourceRequest,
TaskResourceRequests as TaskResourceRequests,
)
-from typing import overload, Dict, Optional, Union
Review comment:
why change this ?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -76,7 +76,7 @@ class MicroBatchExecution(
// transformation is responsible for replacing attributes with their final values.
val disabledSources =
- sparkSession.sqlContext.conf.disabledV2StreamingMicroBatchReaders.split(",")
+ Utils.stringToSeq(sparkSession.sqlContext.conf.disabledV2StreamingMicroBatchReaders)
Review comment:
why change this ?
##########
File path: project/SparkBuild.scala
##########
@@ -213,17 +213,23 @@ object SparkBuild extends PomBuild {
"gcs-maven-central-mirror" at "https://maven-central.storage-download.googleapis.com/maven2/",
DefaultMavenRepository,
Resolver.mavenLocal,
- Resolver.file("local", file(Path.userHome.absolutePath + "/.ivy2/local"))(Resolver.ivyStylePatterns)
+ Resolver.file("ivyLocal", file(Path.userHome.absolutePath + "/.ivy2/local"))(Resolver.ivyStylePatterns)
),
externalResolvers := resolvers.value,
otherResolvers := SbtPomKeys.mvnLocalRepository(dotM2 => Seq(Resolver.file("dotM2", dotM2))).value,
publishLocalConfiguration in MavenCompile := PublishConfiguration()
.withResolverName("dotM2")
.withArtifacts(packagedArtifacts.value.toVector)
.withLogging(ivyLoggingLevel.value),
+ publishLocalConfiguration in SbtCompile := PublishConfiguration()
+ .withResolverName("ivyLocal")
+ .withArtifacts(packagedArtifacts.value.toVector)
+ .withLogging(ivyLoggingLevel.value),
publishMavenStyle in MavenCompile := true,
+ publishMavenStyle in SbtCompile := false,
publishLocal in MavenCompile := publishTask(publishLocalConfiguration in MavenCompile).value,
- publishLocalBoth := Seq(publishLocal in MavenCompile, publishLocal).dependOn.value,
+ publishLocal in SbtCompile := publishTask(publishLocalConfiguration in SbtCompile).value,
+ publishLocal := Seq(publishLocal in MavenCompile, publishLocal in SbtCompile).dependOn.value,
Review comment:
why change this ?
##########
File path: project/plugins.sbt
##########
@@ -27,8 +27,6 @@ addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.15.0")
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.2.4")
-addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.9.2")
-
Review comment:
why change this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706484526
**[Test build #129613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129613/testReport)** for PR 29998 at commit [`a2a4f45`](https://github.com/apache/spark/commit/a2a4f454d9a52f1970c3f5b9909996ffc685079f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706484526
**[Test build #129613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129613/testReport)** for PR 29998 at commit [`a2a4f45`](https://github.com/apache/spark/commit/a2a4f454d9a52f1970c3f5b9909996ffc685079f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706508699
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34223/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706490162
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706503709
**[Test build #129619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129619/testReport)** for PR 29998 at commit [`4fba526`](https://github.com/apache/spark/commit/4fba5264b2c7b8d0da107867e6f218514b3e819a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706495793
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34218/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706490158
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34216/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706490162
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #29998:
URL: https://github.com/apache/spark/pull/29998
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706512002
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706516574
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34224/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706511925
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34223/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706496666
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706489472
**[Test build #129615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129615/testReport)** for PR 29998 at commit [`668778c`](https://github.com/apache/spark/commit/668778cb0b090b4b6e815479f76821b04d3c56ea).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706516580
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706511895
**[Test build #129619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129619/testReport)** for PR 29998 at commit [`4fba526`](https://github.com/apache/spark/commit/4fba5264b2c7b8d0da107867e6f218514b3e819a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706489472
**[Test build #129615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129615/testReport)** for PR 29998 at commit [`668778c`](https://github.com/apache/spark/commit/668778cb0b090b4b6e815479f76821b04d3c56ea).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706508144
**[Test build #129620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129620/testReport)** for PR 29998 at commit [`f756d9c`](https://github.com/apache/spark/commit/f756d9cbf6764ea51143fd928ad5a1a1da985426).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706822941
@zero323 Thank you so much. I involved other commits here by mistake. I will resend a new PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706503709
**[Test build #129619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129619/testReport)** for PR 29998 at commit [`4fba526`](https://github.com/apache/spark/commit/4fba5264b2c7b8d0da107867e6f218514b3e819a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706516757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706496578
**[Test build #129615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129615/testReport)** for PR 29998 at commit [`668778c`](https://github.com/apache/spark/commit/668778cb0b090b4b6e815479f76821b04d3c56ea).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706516757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706516580
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706511933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706511933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706488176
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706585945
Thanks for the changes @zhengruifeng
Could also modify `shared.pyi` and add the import?
```
diff --git a/python/pyspark/ml/classification.pyi b/python/pyspark/ml/classification.pyi
index 74d11cfd11..241f5baf8d 100644
--- a/python/pyspark/ml/classification.pyi
+++ b/python/pyspark/ml/classification.pyi
@@ -26,6 +26,7 @@ from pyspark.ml.base import _PredictorParams
from pyspark.ml.param.shared import (
HasAggregationDepth,
HasBlockSize,
+ HasBlockSizeInMB,
HasElasticNetParam,
HasFitIntercept,
HasMaxIter,
diff --git a/python/pyspark/ml/param/shared.pyi b/python/pyspark/ml/param/shared.pyi
index 5999c0eaa4..bbb4890455 100644
--- a/python/pyspark/ml/param/shared.pyi
+++ b/python/pyspark/ml/param/shared.pyi
@@ -185,3 +185,8 @@ class HasBlockSize(Params):
blockSize: Param[int]
def __init__(self) -> None: ...
def getBlockSize(self) -> int: ...
+
+class HasBlockSizeInMB(Params):
+ blockSizeInMB: Param[float]
+ def __init__(self) -> None: ...
+ def getBlockSizeInMB(self) -> float: ...
```
In general you can verify consistency of annotations locally (sorry, still working on the pipeline):
```bash
# Checks annotations
mypy --no-incremental --config python/mypy.ini python/pyspark
# Checks Spark examples
MYPYPATH=python/ mypy --no-incremental --config python/mypy.ini examples/src/main/python/ml examples/src/main/python/sql examples/src/main/python/sql/streaming
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706488138
**[Test build #129613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129613/testReport)** for PR 29998 at commit [`a2a4f45`](https://github.com/apache/spark/commit/a2a4f454d9a52f1970c3f5b9909996ffc685079f).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait HasBlockSizeInMB extends Params `
* `class HasBlockSizeInMB(Params):`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706508144
**[Test build #129620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129620/testReport)** for PR 29998 at commit [`f756d9c`](https://github.com/apache/spark/commit/f756d9cbf6764ea51143fd928ad5a1a1da985426).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706495796
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706516655
**[Test build #129620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129620/testReport)** for PR 29998 at commit [`f756d9c`](https://github.com/apache/spark/commit/f756d9cbf6764ea51143fd928ad5a1a1da985426).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #29998: [SPARK-32907][ML] adaptively blockify instances - LinearSVC
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #29998:
URL: https://github.com/apache/spark/pull/29998#issuecomment-706517228
also ping @WeichenXu123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org