You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ludatabricks <gi...@git.apache.org> on 2018/05/01 06:25:35 UTC
[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...
GitHub user ludatabricks opened a pull request:
https://github.com/apache/spark/pull/21204
[SPARK-24132][ML]Expand instrumentation for classification
## What changes were proposed in this pull request?
- Add OptionalInstrumentation as argument for getNumClasses in ml.classification.Classifier
- Change the function call for getNumClasses in train() in ml.classification.DecisionTreeClassifier, ml.classification.RandomForestClassifier, and ml.classification.NaiveBayes
- Modify the instrumentation creation in ml.classification.LinearSVC
- Change the log call in ml.classification.OneVsRest and ml.classification.LinearSVC
## How was this patch tested?
Manual.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ludatabricks/spark-1 SPARK-23686
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21204.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21204
----
commit 7b75ed6aaa21faf68f5f2db04eaf550f2d468542
Author: Lu WANG <lu...@...>
Date: 2018-05-01T06:13:36Z
Expand instrumentation for classification
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90223/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21204
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90223/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by ludatabricks <gi...@git.apache.org>.
Github user ludatabricks commented on the issue:
https://github.com/apache/spark/pull/21204
LGTM Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/21204#discussion_r185325228
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala ---
@@ -97,9 +97,10 @@ class DecisionTreeClassifier @Since("1.4.0") (
override def setSeed(value: Long): this.type = set(seed, value)
override protected def train(dataset: Dataset[_]): DecisionTreeClassificationModel = {
+ val instr = Instrumentation.create(this, dataset)
val categoricalFeatures: Map[Int, Int] =
MetadataUtils.getCategoricalFeatures(dataset.schema($(featuresCol)))
- val numClasses: Int = getNumClasses(dataset)
+ val numClasses: Int = getNumClasses(dataset, instr = OptionalInstrumentation.create(instr))
--- End diff --
Also see what else you can log easily (e.g., numFeatures)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90070/testReport)** for PR 21204 at commit [`893c93c`](https://github.com/apache/spark/commit/893c93c9dcdcd674352b09bd370bb19e7e18ca29).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90148/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90076/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90076/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90148/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the issue:
https://github.com/apache/spark/pull/21204
LGTM. Merged into master. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #4163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4163/testReport)** for PR 21204 at commit [`7b75ed6`](https://github.com/apache/spark/commit/7b75ed6aaa21faf68f5f2db04eaf550f2d468542).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90076/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90148/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/21204#discussion_r185325179
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala ---
@@ -97,9 +97,10 @@ class DecisionTreeClassifier @Since("1.4.0") (
override def setSeed(value: Long): this.type = set(seed, value)
override protected def train(dataset: Dataset[_]): DecisionTreeClassificationModel = {
+ val instr = Instrumentation.create(this, dataset)
val categoricalFeatures: Map[Int, Int] =
MetadataUtils.getCategoricalFeatures(dataset.schema($(featuresCol)))
- val numClasses: Int = getNumClasses(dataset)
+ val numClasses: Int = getNumClasses(dataset, instr = OptionalInstrumentation.create(instr))
--- End diff --
In cases like this, you can use instr.logNumClasses() instead of relying on logging within the getNumClasses() method.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21204: [SPARK-24132][ML] Instrumentation improvement for...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21204
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #4163 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4163/testReport)** for PR 21204 at commit [`7b75ed6`](https://github.com/apache/spark/commit/7b75ed6aaa21faf68f5f2db04eaf550f2d468542).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90223/testReport)** for PR 21204 at commit [`e41869e`](https://github.com/apache/spark/commit/e41869ec2c0060f5ed345fdda463ad7f9da70434).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21204
**[Test build #90070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90070/testReport)** for PR 21204 at commit [`893c93c`](https://github.com/apache/spark/commit/893c93c9dcdcd674352b09bd370bb19e7e18ca29).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML]Expand instrumentation for classificati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90070/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21204: [SPARK-24132][ML]Expand instrumentation for class...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/21204#discussion_r185324974
--- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala ---
@@ -103,7 +103,10 @@ abstract class Classifier[
* @throws IllegalArgumentException if metadata does not specify numClasses, and the
* actual numClasses exceeds maxNumClasses
*/
- protected def getNumClasses(dataset: Dataset[_], maxNumClasses: Int = 100): Int = {
+ protected def getNumClasses(dataset: Dataset[_],
--- End diff --
Since we don't have Instrumentation readily available, I recommend using the old logging here. (That will also avoid this API-breaking change which is causing the MiMA failure.)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21204: [SPARK-24132][ML] Instrumentation improvement for classi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21204
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org