You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/04/15 15:05:40 UTC
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/12417
[SPARK-14628][CORE] Simplify task metrics by always tracking read/write metrics
## What changes were proposed in this pull request?
Part of the reason why TaskMetrics and its callers are complicated are due to the optional metrics we collect, including input, output, shuffle read, and shuffle write. I think we can always track them and just assign 0 as the initial values. It is usually very obvious whether a task is supposed to read any data or not. By always tracking them, we can remove a lot of map, foreach, flatMap, getOrElse(0L) calls throughout Spark.
This patch also changes a few behaviors.
1. Removed the distinction of data read/write methods (e.g. Hadoop, Memory, Network, etc).
2. Accumulate all data reads and writes, rather than only the first method. (Fixes SPARK-5225)
## How was this patch tested?
existing tests.
This is bases on https://github.com/apache/spark/pull/12388, with more test fixes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark metrics-refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12417.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12417
----
commit 997f1e1ebdf1f376edd9ade8fe44cbea57a706b5
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-14T07:51:14Z
Always track ShuffleReadMetrics (i.e. not an option)
commit 876b471447981faaf29860a46191af726d4b6777
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-14T22:34:27Z
first round commits
commit b27b97b2bcdc9906a0a1a6d9f33a07865bc427d8
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-14T22:47:40Z
mima and remove more options
commit 4a4a8bffc3712140f619c16e222b97e3e86a52de
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-14T22:51:13Z
remove more options in StatsReportListener
commit 3cac11fee7c3af5200c1decb47d501edd68ee397
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-14T23:04:49Z
fix comment
commit b0493f1e2845e64a7644e7792d3d1c40c2ca9870
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-15T00:54:14Z
fix one test problem
commit 7791215509e9d48c822bcd392cdbb7bea9f8647e
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-15T00:56:09Z
fix more test cases
commit fa78b5e359b239d919b5728970a1467a674dee99
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-04-15T01:17:50Z
delete a failing test case since it is invalid now
commit cbc154f441e37263651c3f074ab7730636ba5115
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-04-15T13:03:02Z
fix all tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210459106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55926/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210548559
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55931/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210548202
**[Test build #55931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55931/consoleFull)** for PR 12417 at commit [`2711075`](https://github.com/apache/spark/commit/27110758fc2b90f40117ba514991c8922fb1b07e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210459094
**[Test build #55926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55926/consoleFull)** for PR 12417 at commit [`cbc154f`](https://github.com/apache/spark/commit/cbc154f441e37263651c3f074ab7730636ba5115).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210669070
**[Test build #2793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2793/consoleFull)** for PR 12417 at commit [`2711075`](https://github.com/apache/spark/commit/27110758fc2b90f40117ba514991c8922fb1b07e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210624458
**[Test build #2793 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2793/consoleFull)** for PR 12417 at commit [`2711075`](https://github.com/apache/spark/commit/27110758fc2b90f40117ba514991c8922fb1b07e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210508367
**[Test build #55931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55931/consoleFull)** for PR 12417 at commit [`2711075`](https://github.com/apache/spark/commit/27110758fc2b90f40117ba514991c8922fb1b07e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210633036
**[Test build #2791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2791/consoleFull)** for PR 12417 at commit [`2711075`](https://github.com/apache/spark/commit/27110758fc2b90f40117ba514991c8922fb1b07e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12417
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210458783
**[Test build #55926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55926/consoleFull)** for PR 12417 at commit [`cbc154f`](https://github.com/apache/spark/commit/cbc154f441e37263651c3f074ab7730636ba5115).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210548556
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210459103
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210671840
Going to merge this first. We can fix the test thing later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210458487
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12417#issuecomment-210593037
**[Test build #2791 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2791/consoleFull)** for PR 12417 at commit [`2711075`](https://github.com/apache/spark/commit/27110758fc2b90f40117ba514991c8922fb1b07e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12417#discussion_r59871010
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/AllStagesResource.scala ---
@@ -250,44 +266,62 @@ private[v1] object AllStagesResource {
resultSerializationTime = internal.resultSerializationTime,
memoryBytesSpilled = internal.memoryBytesSpilled,
diskBytesSpilled = internal.diskBytesSpilled,
- inputMetrics = internal.inputMetrics.map { convertInputMetrics },
- outputMetrics = Option(internal.outputMetrics).flatten.map { convertOutputMetrics },
- shuffleReadMetrics = internal.shuffleReadMetrics.map { convertShuffleReadMetrics },
- shuffleWriteMetrics = internal.shuffleWriteMetrics.map { convertShuffleWriteMetrics }
+ inputMetrics = convertInputMetrics(internal.inputMetrics),
+ outputMetrics = convertOutputMetrics(internal.outputMetrics),
+ shuffleReadMetrics = convertShuffleReadMetrics(internal.shuffleReadMetrics),
+ shuffleWriteMetrics = convertShuffleWriteMetrics(internal.shuffleWriteMetrics)
)
}
- def convertInputMetrics(internal: InternalInputMetrics): InputMetrics = {
- new InputMetrics(
- bytesRead = internal.bytesRead,
- recordsRead = internal.recordsRead
- )
+ def convertInputMetrics(internal: InternalInputMetrics): Option[InputMetrics] = {
+ if (internal.isUpdated) {
+ Some(new InputMetrics(
+ bytesRead = internal.bytesRead,
+ recordsRead = internal.recordsRead
+ ))
+ } else {
+ None
+ }
}
- def convertOutputMetrics(internal: InternalOutputMetrics): OutputMetrics = {
- new OutputMetrics(
- bytesWritten = internal.bytesWritten,
- recordsWritten = internal.recordsWritten
- )
+ def convertOutputMetrics(internal: InternalOutputMetrics): Option[OutputMetrics] = {
+ if (internal.isUpdated) {
+ Some(new OutputMetrics(
+ bytesWritten = internal.bytesWritten,
+ recordsWritten = internal.recordsWritten
+ ))
+ } else {
+ None
+ }
}
- def convertShuffleReadMetrics(internal: InternalShuffleReadMetrics): ShuffleReadMetrics = {
- new ShuffleReadMetrics(
- remoteBlocksFetched = internal.remoteBlocksFetched,
- localBlocksFetched = internal.localBlocksFetched,
- fetchWaitTime = internal.fetchWaitTime,
- remoteBytesRead = internal.remoteBytesRead,
- totalBlocksFetched = internal.totalBlocksFetched,
- recordsRead = internal.recordsRead
- )
+ def convertShuffleReadMetrics(
+ internal: InternalShuffleReadMetrics): Option[ShuffleReadMetrics] = {
+ if (internal.isUpdated) {
+ Some(new ShuffleReadMetrics(
+ remoteBlocksFetched = internal.remoteBlocksFetched,
+ localBlocksFetched = internal.localBlocksFetched,
+ fetchWaitTime = internal.fetchWaitTime,
+ remoteBytesRead = internal.remoteBytesRead,
+ totalBlocksFetched = internal.totalBlocksFetched,
+ recordsRead = internal.recordsRead
--- End diff --
Looks like the internal and external shuffle read metrics don't match, the `localBytesRead` is missing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14628][CORE] Simplify task metrics by a...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12417#discussion_r59923303
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -326,33 +326,35 @@ private[spark] object JsonProtocol {
}
def taskMetricsToJson(taskMetrics: TaskMetrics): JValue = {
- val shuffleReadMetrics: JValue =
- taskMetrics.shuffleReadMetrics.map { rm =>
- ("Remote Blocks Fetched" -> rm.remoteBlocksFetched) ~
- ("Local Blocks Fetched" -> rm.localBlocksFetched) ~
- ("Fetch Wait Time" -> rm.fetchWaitTime) ~
- ("Remote Bytes Read" -> rm.remoteBytesRead) ~
- ("Local Bytes Read" -> rm.localBytesRead) ~
- ("Total Records Read" -> rm.recordsRead)
- }.getOrElse(JNothing)
- val shuffleWriteMetrics: JValue =
- taskMetrics.shuffleWriteMetrics.map { wm =>
- ("Shuffle Bytes Written" -> wm.bytesWritten) ~
- ("Shuffle Write Time" -> wm.writeTime) ~
- ("Shuffle Records Written" -> wm.recordsWritten)
- }.getOrElse(JNothing)
- val inputMetrics: JValue =
- taskMetrics.inputMetrics.map { im =>
- ("Data Read Method" -> im.readMethod.toString) ~
- ("Bytes Read" -> im.bytesRead) ~
- ("Records Read" -> im.recordsRead)
- }.getOrElse(JNothing)
- val outputMetrics: JValue =
- taskMetrics.outputMetrics.map { om =>
- ("Data Write Method" -> om.writeMethod.toString) ~
- ("Bytes Written" -> om.bytesWritten) ~
- ("Records Written" -> om.recordsWritten)
- }.getOrElse(JNothing)
+ val shuffleReadMetrics: JValue = if (taskMetrics.shuffleReadMetrics.isUpdated) {
+ ("Remote Blocks Fetched" -> taskMetrics.shuffleReadMetrics.remoteBlocksFetched) ~
+ ("Local Blocks Fetched" -> taskMetrics.shuffleReadMetrics.localBlocksFetched) ~
+ ("Fetch Wait Time" -> taskMetrics.shuffleReadMetrics.fetchWaitTime) ~
+ ("Remote Bytes Read" -> taskMetrics.shuffleReadMetrics.remoteBytesRead) ~
+ ("Local Bytes Read" -> taskMetrics.shuffleReadMetrics.localBytesRead) ~
+ ("Total Records Read" -> taskMetrics.shuffleReadMetrics.recordsRead)
+ } else {
+ JNothing
+ }
+ val shuffleWriteMetrics: JValue = if (taskMetrics.shuffleWriteMetrics.isUpdated) {
--- End diff --
can't we always output the metrics, and just fix the json protocol test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org