You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jisookim0513 <gi...@git.apache.org> on 2015/12/09 01:57:01 UTC
[GitHub] spark pull request: add cpu time to metrics
GitHub user jisookim0513 opened a pull request:
https://github.com/apache/spark/pull/10212
add cpu time to metrics
Currently task metrics don't support executor CPU time, so there's no way to calculate how much CPU time a stage/task took from History Server metrics. This PR enables reporting CPU time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jisookim0513/spark add-cpu-time-metric
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10212.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10212
----
commit 30752cb9b3e91366fe2ac16ca769e8fc7e8dcf54
Author: jisookim <ji...@gmail.com>
Date: 2015-12-08T23:44:39Z
add cpu time to metrics
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64096/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10212
Failure looks unrelated... retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10212
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/10212
@vanzin I updated the patch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/10212
@vanzin sure will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64099/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65834/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12221] add cpu time to metrics
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10212#issuecomment-163209990
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/10212
@vanzin thanks, I was about to ask for a retest :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10212
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80155278
--- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala ---
@@ -1097,7 +1100,9 @@ private[spark] object JsonProtocolSuite extends Assertions {
| },
| "Task Metrics": {
| "Executor Deserialize Time": 300,
+ | "Executor Deserialize CPU Time": 0,
--- End diff --
AFAIK, JsonProtolSuite creates a JSON string from the event created by `makeTaskMetrics()`:
'makeTaskMetrics(300L, 400L, 500L, 600L, 700, 800, hasHadoopInput = true, hasOutput = false))'.
I tried changing `makeTaskMetrics()` to accept deserialize CPU time and CPU time as arguments , but that ended up violating scalaStyle by having more than 10 parameters..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65815/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10212
@jisookim0513 are you still around? Do you mind updating the patch so we can trigger tests?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65787/consoleFull)** for PR 10212 at commit [`d9c5f8f`](https://github.com/apache/spark/commit/d9c5f8f23b64b948d11fd94823b015e7133da20d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65787/consoleFull)** for PR 10212 at commit [`d9c5f8f`](https://github.com/apache/spark/commit/d9c5f8f23b64b948d11fd94823b015e7133da20d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class SparkContext(config: SparkConf) extends Logging `
* `class ChiSqSelector @Since(\"2.1.0\") () extends Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/10212
@vanzin this PR had passed all tests. Could you merge it if I fix the recently introduced conflicts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80184799
--- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala ---
@@ -1097,7 +1100,9 @@ private[spark] object JsonProtocolSuite extends Assertions {
| },
| "Task Metrics": {
| "Executor Deserialize Time": 300,
+ | "Executor Deserialize CPU Time": 0,
--- End diff --
Yeah I tested it on my testing cluster, but this makes sense. I will add non-zero CPU times by setting the CPU times same as given wall times.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #64099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64099/consoleFull)** for PR 10212 at commit [`5a9ea8e`](https://github.com/apache/spark/commit/5a9ea8e2c2d9562af75baffe5eab735e78f1a8ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10212
Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80156386
--- Diff: core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json ---
@@ -6,6 +6,7 @@
"numCompleteTasks" : 8,
"numFailedTasks" : 0,
"executorRunTime" : 162,
+ "executorCpuTime" : 0,
--- End diff --
Hmm I thought HistoryServerSuite runs with included log files (that don't have CPU time). So this is an expected result since those logs don't have cpu time fields.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/10212
@vanzin could you merge this? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #64085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64085/consoleFull)** for PR 10212 at commit [`bd19098`](https://github.com/apache/spark/commit/bd19098e1c08f09af33598b65900db54e8aa07b6).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #64085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64085/consoleFull)** for PR 10212 at commit [`bd19098`](https://github.com/apache/spark/commit/bd19098e1c08f09af33598b65900db54e8aa07b6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65787/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80137728
--- Diff: core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json ---
@@ -6,6 +6,7 @@
"numCompleteTasks" : 8,
"numFailedTasks" : 0,
"executorRunTime" : 162,
+ "executorCpuTime" : 0,
--- End diff --
Do you need to add these values to all these files? The code should be able to handle the old logs that don't have the value, and not adding these would be a good test case for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12221] add cpu time to metrics
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r47083416
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -718,6 +719,7 @@ private[spark] object JsonProtocol {
metrics.setHostname((json \ "Host Name").extract[String])
metrics.setExecutorDeserializeTime((json \ "Executor Deserialize Time").extract[Long])
metrics.setExecutorRunTime((json \ "Executor Run Time").extract[Long])
+ metrics.setExecutorCpuTime((json \ "Executor CPU Time").extract[Long])
--- End diff --
what happens when the history server is tries to deserialise a history from an earlier spark version, one which doesn't have a CPU time? As it looks to me through my scan through the code that this is going to fail.
HistoryServerSuite is the regression test here —it does have job histories without the relevant metric.
It would benefit from having another reference test run here for playback
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #64096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64096/consoleFull)** for PR 10212 at commit [`bf8d4f8`](https://github.com/apache/spark/commit/bf8d4f8ceaf32b56682642132ae5b6efe58e58d5).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65834 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65834/consoleFull)** for PR 10212 at commit [`f0ef503`](https://github.com/apache/spark/commit/f0ef503f9e732b91d405d1e15dade58e78999052).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65679/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10212
Sure. Just remember to ping someone, otherwise things get lost in the mountain of e-mails github generates.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10212#issuecomment-163073074
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65679/consoleFull)** for PR 10212 at commit [`d4d6f76`](https://github.com/apache/spark/commit/d4d6f76da4b2e94b9b97c512ffc8e8b42414a675).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #64099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64099/consoleFull)** for PR 10212 at commit [`5a9ea8e`](https://github.com/apache/spark/commit/5a9ea8e2c2d9562af75baffe5eab735e78f1a8ca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80137957
--- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala ---
@@ -1097,7 +1100,9 @@ private[spark] object JsonProtocolSuite extends Assertions {
| },
| "Task Metrics": {
| "Executor Deserialize Time": 300,
+ | "Executor Deserialize CPU Time": 0,
--- End diff --
Could you use values other than `0` so that you're sure the code is actually parsing the value, instead of falling into the default case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65686/consoleFull)** for PR 10212 at commit [`2ff5bdc`](https://github.com/apache/spark/commit/2ff5bdce904f4ca55096ea4616557342003b8043).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80165213
--- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala ---
@@ -1097,7 +1100,9 @@ private[spark] object JsonProtocolSuite extends Assertions {
| },
| "Task Metrics": {
| "Executor Deserialize Time": 300,
+ | "Executor Deserialize CPU Time": 0,
--- End diff --
I still think it would be worth to fix this; just find some way around the style check.
Otherwise, you're not really testing whether the parsing code is actually parsing the field (what if there's a typo somewhere?).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65686/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65686/consoleFull)** for PR 10212 at commit [`2ff5bdc`](https://github.com/apache/spark/commit/2ff5bdce904f4ca55096ea4616557342003b8043).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80184744
--- Diff: core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json ---
@@ -6,6 +6,7 @@
"numCompleteTasks" : 8,
"numFailedTasks" : 0,
"executorRunTime" : 162,
+ "executorCpuTime" : 0,
--- End diff --
Oh no, these are expected outputs. I think the inputs are stored under `src/test/resources/spark-events`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80165121
--- Diff: core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json ---
@@ -6,6 +6,7 @@
"numCompleteTasks" : 8,
"numFailedTasks" : 0,
"executorRunTime" : 162,
+ "executorCpuTime" : 0,
--- End diff --
Oh, I though these were input to the history server, not "golden files" that it checks against... if that's the case, ignore my comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/10212
@vanzin thanks a lot!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65834/consoleFull)** for PR 10212 at commit [`f0ef503`](https://github.com/apache/spark/commit/f0ef503f9e732b91d405d1e15dade58e78999052).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80137783
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -759,7 +761,15 @@ private[spark] object JsonProtocol {
return metrics
}
metrics.setExecutorDeserializeTime((json \ "Executor Deserialize Time").extract[Long])
+ metrics.setExecutorDeserializeCpuTime((json \ "Executor Deserialize CPU Time") match {
+ case JNothing => 0
+ case x => x.extract[Long]}
+ )
metrics.setExecutorRunTime((json \ "Executor Run Time").extract[Long])
+ metrics.setExecutorCpuTime((json \ "Executor CPU Time") match {
+ case JNothing => 0
+ case x => x.extract[Long]}
--- End diff --
nit: move '}' to next line (with the ')'). Also above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65815/consoleFull)** for PR 10212 at commit [`f0ef503`](https://github.com/apache/spark/commit/f0ef503f9e732b91d405d1e15dade58e78999052).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r80156532
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -759,7 +761,15 @@ private[spark] object JsonProtocol {
return metrics
}
metrics.setExecutorDeserializeTime((json \ "Executor Deserialize Time").extract[Long])
+ metrics.setExecutorDeserializeCpuTime((json \ "Executor Deserialize CPU Time") match {
+ case JNothing => 0
+ case x => x.extract[Long]}
+ )
metrics.setExecutorRunTime((json \ "Executor Run Time").extract[Long])
+ metrics.setExecutorCpuTime((json \ "Executor CPU Time") match {
+ case JNothing => 0
+ case x => x.extract[Long]}
--- End diff --
Will fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65679 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65679/consoleFull)** for PR 10212 at commit [`d4d6f76`](https://github.com/apache/spark/commit/d4d6f76da4b2e94b9b97c512ffc8e8b42414a675).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10212#issuecomment-206511863
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64085/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12221] add cpu time to metrics
Posted by jisookim0513 <gi...@git.apache.org>.
Github user jisookim0513 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10212#discussion_r54951079
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -718,6 +719,7 @@ private[spark] object JsonProtocol {
metrics.setHostname((json \ "Host Name").extract[String])
metrics.setExecutorDeserializeTime((json \ "Executor Deserialize Time").extract[Long])
metrics.setExecutorRunTime((json \ "Executor Run Time").extract[Long])
+ metrics.setExecutorCpuTime((json \ "Executor CPU Time").extract[Long])
--- End diff --
Yeah it won't be able to deserialize a history from an earlier version. Would it be better to make this backward-compatible? (Sorry for the super late response)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12221] add cpu time to metrics
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/10212#issuecomment-163210092
(I may be trusted enough to start a run..let's see)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10212
LGTM pending tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #64096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64096/consoleFull)** for PR 10212 at commit [`bf8d4f8`](https://github.com/apache/spark/commit/bf8d4f8ceaf32b56682642132ae5b6efe58e58d5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10212
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10212
@jisookim0513 unfortunately there are conflicts once more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/10212
**[Test build #65815 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65815/consoleFull)** for PR 10212 at commit [`f0ef503`](https://github.com/apache/spark/commit/f0ef503f9e732b91d405d1e15dade58e78999052).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org