You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shahidki31 <gi...@git.apache.org> on 2018/09/26 07:14:14 UTC
[GitHub] spark pull request #22555: [SPARK-25536][CORE][Minor]metric value for METRIC...
GitHub user shahidki31 opened a pull request:
https://github.com/apache/spark/pull/22555
[SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT_RECORDS_WRITTEN is incorrect
## What changes were proposed in this pull request?
changed metric value of METRIC_OUTPUT_RECORDS_WRITTEN from 'task.metrics.inputMetrics.recordsRead' to 'outputMetrics.recordsWritten'
## How was this patch tested?
Existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shahidki31/spark SPARK-25536
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22555.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22555
----
commit 332adef866f27efd615fb0b9a9717ab734b7377d
Author: Shahid <sh...@...>
Date: 2018-09-26T05:44:33Z
[SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORDS_WRITTEN is incorrect
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96646/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/22555
It is random failure. passed in local. Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96638 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96638/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22555: [SPARK-25536][CORE][Minor]metric value for METRIC...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22555#discussion_r220656095
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -464,7 +464,7 @@ private[spark] class Executor(
executorSource.METRIC_OUTPUT_BYTES_WRITTEN
.inc(task.metrics.outputMetrics.bytesWritten)
executorSource.METRIC_OUTPUT_RECORDS_WRITTEN
- .inc(task.metrics.inputMetrics.recordsRead)
+ .inc(task.metrics.outputMetrics.recordsWritten)
--- End diff --
Could you remove `[MINOR]`? Actually, SPARK-25536 issue is marked as 'MAJOR' bug.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22555
Ur, @wangyum . https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96646/ is triggered already for that hanged `Test build #96638`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96649/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96638/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96649/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/22555
cc @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22555
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22555
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96646/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22555: [SPARK-25536][CORE][Minor]metric value for METRIC...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22555#discussion_r220657964
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -464,7 +464,7 @@ private[spark] class Executor(
executorSource.METRIC_OUTPUT_BYTES_WRITTEN
.inc(task.metrics.outputMetrics.bytesWritten)
executorSource.METRIC_OUTPUT_RECORDS_WRITTEN
- .inc(task.metrics.inputMetrics.recordsRead)
+ .inc(task.metrics.outputMetrics.recordsWritten)
--- End diff --
In addition, please add more description about when this bug is introduced. e.g. `SPARK-22190` in this case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22555
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/22555
Retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/22555
Sorry @dongjoon-hyun
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96646/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22555
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22555
The test hangs and fails, but it's irrelevant again.
```scala
[info] DistributedSuite:
...
9/26/18 3:10:51 PM =[info] - recover from node failures with replication *** FAILED *** (2 hours, 46 minutes, 24 seconds)
[info] org.apache.spark.SparkException: Job 10 cancelled as part of cancellation of all jobs
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22555
Thank you, @shahidki31 , @cloud-fan , @wangyum .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96626/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22555#discussion_r220743860
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -464,7 +464,7 @@ private[spark] class Executor(
executorSource.METRIC_OUTPUT_BYTES_WRITTEN
.inc(task.metrics.outputMetrics.bytesWritten)
executorSource.METRIC_OUTPUT_RECORDS_WRITTEN
- .inc(task.metrics.inputMetrics.recordsRead)
+ .inc(task.metrics.outputMetrics.recordsWritten)
--- End diff --
Thanks @dongjoon-hyun . I have updated the PR description.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22555
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96649/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96638/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22555
Yep. The failure looks irrelevant to this.
```
org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.HDFSMetadataLog: metadata directory collision
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22555
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96626/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22555
**[Test build #96626 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96626/testReport)** for PR 22555 at commit [`332adef`](https://github.com/apache/spark/commit/332adef866f27efd615fb0b9a9717ab734b7377d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE]metric value for METRIC_OUTPUT_RECORD...
Posted by shahidki31 <gi...@git.apache.org>.
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/22555
Thank you @dongjoon-hyun for merging
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22555: [SPARK-25536][CORE][Minor]metric value for METRIC_OUTPUT...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22555
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org