You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mgaido91 <gi...@git.apache.org> on 2018/01/08 22:39:29 UTC
[GitHub] spark pull request #20189: [SPARK-22975] MetricsReporter should not throw ex...
GitHub user mgaido91 opened a pull request:
https://github.com/apache/spark/pull/20189
[SPARK-22975] MetricsReporter should not throw exception when there was no progress reported
## What changes were proposed in this pull request?
`MetricsReporter ` assumes that there has been some progress for the query, ie. `lastProgress` is not null. If this is not true, as it might happen in particular conditions, a `NullPointerException` can be thrown.
The PR checks whether there is a `lastProgress` and if this is not true, it returns a default value for the metrics.
## How was this patch tested?
added UT
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mgaido91/spark SPARK-22975
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20189.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20189
----
commit 1185a513bd807df03a24b22370903d17301fd415
Author: Marco Gaido <ma...@...>
Date: 2018-01-08T22:28:12Z
[SPARK-22975] MetricsReporter should not throw exception when there was no progress reported
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20189: [SPARK-22975] MetricsReporter should not throw ex...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20189#discussion_r160829638
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetricsReporter.scala ---
@@ -39,14 +35,18 @@ class MetricsReporter(
// Metric names should not have . in them, so that all the metrics of a query are identified
// together in Ganglia as a single metric group
- registerGauge("inputRate-total", () => stream.lastProgress.inputRowsPerSecond)
- registerGauge("processingRate-total", () => stream.lastProgress.processedRowsPerSecond)
- registerGauge("latency", () => stream.lastProgress.durationMs.get("triggerExecution").longValue())
-
- private def registerGauge[T](name: String, f: () => T)(implicit num: Numeric[T]): Unit = {
+ registerGauge("inputRate-total", (lastProgress) => lastProgress.inputRowsPerSecond, 0.0)
+ registerGauge("processingRate-total", (lastProgress) => lastProgress.processedRowsPerSecond, 0.0)
+ registerGauge("latency",
+ (lastProgress) => lastProgress.durationMs.get("triggerExecution").longValue(), 0L)
+
+ private def registerGauge[T](
+ name: String,
+ f: (StreamingQueryProgress) => T,
--- End diff --
nit: `(StreamingQueryProgress) => T` -> `StreamingQueryProgress => T,`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20189
Could you update the PR title to include the component such as `[SPARK-22975][SS] MetricsReporter should not throw exception when there was no progress reported`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20189: [SPARK-22975][SS] MetricsReporter should not thro...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20189
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85810/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20189: [SPARK-22975] MetricsReporter should not throw ex...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20189#discussion_r160829808
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala ---
@@ -424,6 +424,31 @@ class StreamingQuerySuite extends StreamTest with BeforeAndAfter with Logging wi
}
}
+ test("SPARK-22975: MetricsReporter defaults when there was no progress reported") {
+ withSQLConf("spark.sql.streaming.metricsEnabled" -> "true") {
+ BlockingSource.latch = new CountDownLatch(1)
--- End diff --
You can use `memory` source and `console` sink to simplify the codes like this:
```
test("SPARK-22975: MetricsReporter defaults when there was no progress reported") {
withSQLConf("spark.sql.streaming.metricsEnabled" -> "true") {
withTempDir { tempDir =>
val sq = MemoryStream[Int].toDF
.writeStream
.format("console")
.start()
.asInstanceOf[StreamingQueryWrapper]
.streamingQuery
val gauges = sq.streamMetrics.metricRegistry.getGauges
assert(gauges.get("latency").getValue.asInstanceOf[Long] == 0)
assert(gauges.get("processingRate-total").getValue.asInstanceOf[Double] == 0.0)
assert(gauges.get("inputRate-total").getValue.asInstanceOf[Double] == 0.0)
sq.stop()
}
}
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20189
@mgaido91 Ah, sorry. There is a race condition in this test since the query may make progress when we are checking the results. Please revert to your previous test. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20189
@tdas @zsxwing may I kindly ask if you might take a look at this? Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85861/testReport)** for PR 20189 at commit [`b7dc922`](https://github.com/apache/spark/commit/b7dc92235f434ee0630bdb6af918ee8e58d2fa2b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85967/testReport)** for PR 20189 at commit [`c67e07b`](https://github.com/apache/spark/commit/c67e07b66726d57c2b7a3462b226439518d9ebed).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86030/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85850/testReport)** for PR 20189 at commit [`7242eab`](https://github.com/apache/spark/commit/7242eabe00ce84cb132a4a4f16cb53bed1e6afa7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #86030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86030/testReport)** for PR 20189 at commit [`bfb706d`](https://github.com/apache/spark/commit/bfb706d481dfa9aa88b58f9a7b8f7da87956c742).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85861/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85967 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85967/testReport)** for PR 20189 at commit [`c67e07b`](https://github.com/apache/spark/commit/c67e07b66726d57c2b7a3462b226439518d9ebed).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20189
LGTM pending tests
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #86030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86030/testReport)** for PR 20189 at commit [`bfb706d`](https://github.com/apache/spark/commit/bfb706d481dfa9aa88b58f9a7b8f7da87956c742).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85861/testReport)** for PR 20189 at commit [`b7dc922`](https://github.com/apache/spark/commit/b7dc92235f434ee0630bdb6af918ee8e58d2fa2b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85967/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85998/testReport)** for PR 20189 at commit [`bfb706d`](https://github.com/apache/spark/commit/bfb706d481dfa9aa88b58f9a7b8f7da87956c742).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20189
@zsxwing the test you suggested fails on Jenkins while locally it passes. I am no sure about the reason since I cannot reproduce it. I had the same test error in my previous trials. Previously the problem was that running other test cases before this one produced the issue. May you please advice what to do? Can I revert to my previous test?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85810/testReport)** for PR 20189 at commit [`1185a51`](https://github.com/apache/spark/commit/1185a513bd807df03a24b22370903d17301fd415).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20189: [SPARK-22975] MetricsReporter should not throw ex...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20189#discussion_r160829571
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetricsReporter.scala ---
@@ -39,14 +35,18 @@ class MetricsReporter(
// Metric names should not have . in them, so that all the metrics of a query are identified
// together in Ganglia as a single metric group
- registerGauge("inputRate-total", () => stream.lastProgress.inputRowsPerSecond)
- registerGauge("processingRate-total", () => stream.lastProgress.processedRowsPerSecond)
- registerGauge("latency", () => stream.lastProgress.durationMs.get("triggerExecution").longValue())
-
- private def registerGauge[T](name: String, f: () => T)(implicit num: Numeric[T]): Unit = {
+ registerGauge("inputRate-total", (lastProgress) => lastProgress.inputRowsPerSecond, 0.0)
+ registerGauge("processingRate-total", (lastProgress) => lastProgress.processedRowsPerSecond, 0.0)
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20189
Thanks! Merging to master, 2.3 and 2.2.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20189: [SPARK-22975] MetricsReporter should not throw ex...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20189#discussion_r160829544
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetricsReporter.scala ---
@@ -39,14 +35,18 @@ class MetricsReporter(
// Metric names should not have . in them, so that all the metrics of a query are identified
// together in Ganglia as a single metric group
- registerGauge("inputRate-total", () => stream.lastProgress.inputRowsPerSecond)
- registerGauge("processingRate-total", () => stream.lastProgress.processedRowsPerSecond)
- registerGauge("latency", () => stream.lastProgress.durationMs.get("triggerExecution").longValue())
-
- private def registerGauge[T](name: String, f: () => T)(implicit num: Numeric[T]): Unit = {
+ registerGauge("inputRate-total", (lastProgress) => lastProgress.inputRowsPerSecond, 0.0)
--- End diff --
nit: `registerGauge("inputRate-total", _.inputRowsPerSecond, 0.0)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85810/testReport)** for PR 20189 at commit [`1185a51`](https://github.com/apache/spark/commit/1185a513bd807df03a24b22370903d17301fd415).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85998/testReport)** for PR 20189 at commit [`bfb706d`](https://github.com/apache/spark/commit/bfb706d481dfa9aa88b58f9a7b8f7da87956c742).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85850/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20189
**[Test build #85850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85850/testReport)** for PR 20189 at commit [`7242eab`](https://github.com/apache/spark/commit/7242eabe00ce84cb132a4a4f16cb53bed1e6afa7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20189: [SPARK-22975] MetricsReporter should not throw ex...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20189#discussion_r160829605
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetricsReporter.scala ---
@@ -39,14 +35,18 @@ class MetricsReporter(
// Metric names should not have . in them, so that all the metrics of a query are identified
// together in Ganglia as a single metric group
- registerGauge("inputRate-total", () => stream.lastProgress.inputRowsPerSecond)
- registerGauge("processingRate-total", () => stream.lastProgress.processedRowsPerSecond)
- registerGauge("latency", () => stream.lastProgress.durationMs.get("triggerExecution").longValue())
-
- private def registerGauge[T](name: String, f: () => T)(implicit num: Numeric[T]): Unit = {
+ registerGauge("inputRate-total", (lastProgress) => lastProgress.inputRowsPerSecond, 0.0)
+ registerGauge("processingRate-total", (lastProgress) => lastProgress.processedRowsPerSecond, 0.0)
+ registerGauge("latency",
+ (lastProgress) => lastProgress.durationMs.get("triggerExecution").longValue(), 0L)
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20189
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85998/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20189: [SPARK-22975][SS] MetricsReporter should not throw excep...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20189
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org