You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/29 09:09:47 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
AngersZhuuuu opened a new pull request #31992:
URL: https://github.com/apache/spark/pull/31992
### What changes were proposed in this pull request?
In current EventLoggingListener, we won't write SparkListenerExecutorMetricsUpdate message at all
```
override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
if (shouldLogStageExecutorMetrics) {
event.executorUpdates.foreach { case (stageKey1, newPeaks) =>
liveStageExecutorMetrics.foreach { case (stageKey2, metricsPerExecutor) =>
// If the update came from the driver, stageKey1 will be the dummy key (-1, -1),
// so record those peaks for all active stages.
// Otherwise, record the peaks for the matching stage.
if (stageKey1 == DRIVER_STAGE_KEY || stageKey1 == stageKey2) {
val metrics = metricsPerExecutor.getOrElseUpdate(
event.execId, new ExecutorMetrics())
metrics.compareAndUpdatePeakValues(newPeaks)
}
}
}
}
}
```
It cause we can't get driver executor peakMemoryMetrics in SHS at all since there's no event about this. We can got executor's peakExecutorMetrics since it will update with TaskEnd events's peakMemoryMetrics.
So in this pr, I add support to send SparkListenerExecutorMetricsUpdateEventLog of `driver`.
### Why are the changes needed?
Make user can got driver's peakMemoryMetrics in SHS.
### Does this PR introduce _any_ user-facing change?
user can got driver's peakMemoryMetrics in SHS.
### How was this patch tested?
Mannul test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822356454
**[Test build #137583 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137583/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862089220
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44371/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862100618
**[Test build #139837 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139837/testReport)** for PR 31992 at commit [`c9a4a67`](https://github.com/apache/spark/commit/c9a4a6789586771720739cd71354029b64e5b358).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862035537
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862094216
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822165685
**[Test build #137559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137559/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862038027
@AngersZhuuuu BTW did you disable GA in your fork repo? It should be enabled so PR leverage the GA resources in your forked repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31992:
URL: https://github.com/apache/spark/pull/31992#discussion_r652346351
##########
File path: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
##########
@@ -618,6 +619,10 @@ class EventLoggingListenerSuite extends SparkFunSuite with LocalSparkContext wit
assert(expected.stageInfo.stageId === actual.stageInfo.stageId)
case (expected: SparkListenerTaskEnd, actual: SparkListenerTaskEnd) =>
assert(expected.stageId === actual.stageId)
+ case (expected: SparkListenerExecutorMetricsUpdate,
+ actual: SparkListenerExecutorMetricsUpdate) =>
Review comment:
```suggestion
actual: SparkListenerExecutorMetricsUpdate) =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809878948
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41262/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862046824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809232049
**[Test build #136645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136645/testReport)** for PR 31992 at commit [`58ace57`](https://github.com/apache/spark/commit/58ace57460eca084249459a8012b0e4a0f9b52af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-814569934
> As I understood based on description and proposed changes, the right level of integration for this would be with metrics subsystem.. While we can do it via other means, it is not optimal to do so.
> Having said that, I dont have strong opinions on this ... will let some more involved with this comment better.
Yea, also ping @srowen @Ngone51 @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809398720
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41227/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809445096
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862105275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862044197
> @AngersZhuuuu BTW did you disable GA in your fork repo? It should be enabled so PR leverage the GA resources in your forked repo.
No, but this pr is a little long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822165685
**[Test build #137559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137559/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809430331
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41227/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862043865
> @AngersZhuuuu, mind making the PR description disambiguous? what's "driver executor peakMemoryMetrics"?
How about current
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809925234
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136680/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31992:
URL: https://github.com/apache/spark/pull/31992#discussion_r603734069
##########
File path: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
##########
@@ -618,6 +619,10 @@ class EventLoggingListenerSuite extends SparkFunSuite with LocalSparkContext wit
assert(expected.stageInfo.stageId === actual.stageInfo.stageId)
case (expected: SparkListenerTaskEnd, actual: SparkListenerTaskEnd) =>
assert(expected.stageId === actual.stageId)
+ case (expected: SparkListenerExecutorMetricsUpdate,
+ acture: SparkListenerExecutorMetricsUpdate) =>
Review comment:
> `acture` -> `actual`
DOne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862044801
**[Test build #139843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139843/testReport)** for PR 31992 at commit [`87a079e`](https://github.com/apache/spark/commit/87a079e5fa159dad343adcf8ac3f158ff4870f6b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809381330
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136646/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809238685
**[Test build #136646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136646/testReport)** for PR 31992 at commit [`936eefd`](https://github.com/apache/spark/commit/936eefd962467136b96754bcaea24da0f8d9ec81).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809234086
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862150590
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44378/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862041476
**[Test build #139842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139842/testReport)** for PR 31992 at commit [`6c81e2d`](https://github.com/apache/spark/commit/6c81e2dd74b7f49b3ba30b8618d1a502db1246dc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809238685
**[Test build #136646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136646/testReport)** for PR 31992 at commit [`936eefd`](https://github.com/apache/spark/commit/936eefd962467136b96754bcaea24da0f8d9ec81).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862046824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822197314
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822197314
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809234086
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862038185
cc @HeartSaVioR too FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809354432
**[Test build #136646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136646/testReport)** for PR 31992 at commit [`936eefd`](https://github.com/apache/spark/commit/936eefd962467136b96754bcaea24da0f8d9ec81).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #31992:
URL: https://github.com/apache/spark/pull/31992#discussion_r603532299
##########
File path: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala
##########
@@ -249,6 +249,9 @@ private[spark] class EventLoggingListener(
}
override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
+ if (event.execId == SparkContext.DRIVER_IDENTIFIER) {
+ logEvent(event)
+ }
Review comment:
Do this only when `shouldLogStageExecutorMetrics` is enabled ?
##########
File path: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
##########
@@ -618,6 +619,10 @@ class EventLoggingListenerSuite extends SparkFunSuite with LocalSparkContext wit
assert(expected.stageInfo.stageId === actual.stageInfo.stageId)
case (expected: SparkListenerTaskEnd, actual: SparkListenerTaskEnd) =>
assert(expected.stageId === actual.stageId)
+ case (expected: SparkListenerExecutorMetricsUpdate,
+ acture: SparkListenerExecutorMetricsUpdate) =>
Review comment:
`acture` -> `actual`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862144812
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139842/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822377548
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137583/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822224403
**[Test build #137583 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137583/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822197277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822266593
**[Test build #137559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137559/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809591074
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41235/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862166587
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862069041
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44371/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822249363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809860209
**[Test build #136680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136680/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862073263
**[Test build #139850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139850/testReport)** for PR 31992 at commit [`87a079e`](https://github.com/apache/spark/commit/87a079e5fa159dad343adcf8ac3f158ff4870f6b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862010798
> I don't know enough to have an opinion on this. I think the key questions are - what is the most consistent thing to do, and, are there any performance problems with adding this information to events?
Since for spark admin, we always want to build system to know the app's running status in our cluster and let user to change the memory configuration if they don't set it reasonable. So we nee to know the peak memory usage. Although we can get this information form metrics system but we need to integrate restful api's metrics data and metrics system's information.
This pr make us can get driver's memory usage from hisitory server's restful api.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862048157
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862041476
**[Test build #139842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139842/testReport)** for PR 31992 at commit [`6c81e2d`](https://github.com/apache/spark/commit/6c81e2dd74b7f49b3ba30b8618d1a502db1246dc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809430331
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41227/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822257601
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862144812
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139842/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809381330
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136646/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862046804
**[Test build #139843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139843/testReport)** for PR 31992 at commit [`87a079e`](https://github.com/apache/spark/commit/87a079e5fa159dad343adcf8ac3f158ff4870f6b).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809925234
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136680/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862021515
**[Test build #139837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139837/testReport)** for PR 31992 at commit [`c9a4a67`](https://github.com/apache/spark/commit/c9a4a6789586771720739cd71354029b64e5b358).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862071619
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-813543189
As I understood based on description and proposed changes, the right level of integration for this would be with metrics subsystem.. While we can do it via other means, it is not optimal to do so.
Having said that, I dont have strong opinions on this ... will let some more involved with this comment better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862036980
Also do you mean `SparkListenerExecutorMetricsUpdateEvent` by `SparkListenerExecutorMetricsUpdateEventLog`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31992:
URL: https://github.com/apache/spark/pull/31992#discussion_r652323059
##########
File path: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala
##########
@@ -249,6 +249,9 @@ private[spark] class EventLoggingListener(
}
override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
+ if (event.execId == SparkContext.DRIVER_IDENTIFIER) {
+ logEvent(event)
+ }
Review comment:
> Currently, we have a single event for both driver and executor metrics update - differentiated by exec id.
> I dont have strong opinions on this, but if we have a flag (`shouldLogStageExecutorMetrics`) controlling whether metrics are to be updated, we should consistently apply it IMO.
@mridulm Follow this comment, how about current.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809445161
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809881727
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41262/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822257601
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809881727
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41262/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809399670
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862186551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809918966
**[Test build #136680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136680/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822267847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862186551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862142922
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809382387
**[Test build #136653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136653/testReport)** for PR 31992 at commit [`7054b58`](https://github.com/apache/spark/commit/7054b5810ad07588c1fd8490d44c6549c46df3e5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822267847
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-811668969
ping @mridulm Any more update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862048157
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809390973
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41227/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809479822
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136653/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862188623
**[Test build #139866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139866/testReport)** for PR 31992 at commit [`87a079e`](https://github.com/apache/spark/commit/87a079e5fa159dad343adcf8ac3f158ff4870f6b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862105275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862166549
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #31992:
URL: https://github.com/apache/spark/pull/31992#discussion_r607236171
##########
File path: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala
##########
@@ -249,6 +249,9 @@ private[spark] class EventLoggingListener(
}
override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
+ if (event.execId == SparkContext.DRIVER_IDENTIFIER) {
+ logEvent(event)
+ }
Review comment:
Currently, we have a single event for both driver and executor metrics update - differentiated by exec id.
I dont have strong opinions on this, but if we have a flag (`shouldLogStageExecutorMetrics`) controlling whether metrics are to be updated, we should consistently apply it IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809860209
**[Test build #136680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136680/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862021515
**[Test build #139837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139837/testReport)** for PR 31992 at commit [`c9a4a67`](https://github.com/apache/spark/commit/c9a4a6789586771720739cd71354029b64e5b358).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862023314
Thanks for the update @AngersZhuuuu
+CC @zhouyejoe, @thejdeep PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-814574407
I don't know enough to have an opinion on this. I think the key questions are - what is the most consistent thing to do, and, are there any performance problems with adding this information to events?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-822224403
**[Test build #137583 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137583/testReport)** for PR 31992 at commit [`02ad9de`](https://github.com/apache/spark/commit/02ad9de7ece49e294c618b4067d3cd5017ba68ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809479822
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136653/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809445161
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809615687
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41235/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809234054
**[Test build #136645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136645/testReport)** for PR 31992 at commit [`58ace57`](https://github.com/apache/spark/commit/58ace57460eca084249459a8012b0e4a0f9b52af).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809307677
cc @Ngone51 FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862044801
**[Test build #139843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139843/testReport)** for PR 31992 at commit [`87a079e`](https://github.com/apache/spark/commit/87a079e5fa159dad343adcf8ac3f158ff4870f6b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862048138
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862069463
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809881705
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41262/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809615687
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41235/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809854277
> Who is consuming this event in spark ?
> Btw, if all you want is monitor jvm stats (outside of spark ui), use codahale integration instead ?
`AppStatusListener` will consumer this
```
override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
```
I know spark metrics frame work can have a monitor on this. But when we want to build an application analysis system like Dr.elephant. We want to get app status from restful api, right? Driver's memory usage is an important index too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862173600
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44378/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809232049
**[Test build #136645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136645/testReport)** for PR 31992 at commit [`58ace57`](https://github.com/apache/spark/commit/58ace57460eca084249459a8012b0e4a0f9b52af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862158957
**[Test build #139850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139850/testReport)** for PR 31992 at commit [`87a079e`](https://github.com/apache/spark/commit/87a079e5fa159dad343adcf8ac3f158ff4870f6b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31992:
URL: https://github.com/apache/spark/pull/31992#discussion_r603734003
##########
File path: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala
##########
@@ -249,6 +249,9 @@ private[spark] class EventLoggingListener(
}
override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
+ if (event.execId == SparkContext.DRIVER_IDENTIFIER) {
+ logEvent(event)
+ }
Review comment:
> Do this only when `shouldLogStageExecutorMetrics` is enabled ?
I don't thinks it should be controlled by `shouldLogStageExecutorMetrics `. since driver's metrics is not related to stage executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862073263
**[Test build #139850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139850/testReport)** for PR 31992 at commit [`87a079e`](https://github.com/apache/spark/commit/87a079e5fa159dad343adcf8ac3f158ff4870f6b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862036119
@AngersZhuuuu, mind making the PR description disambiguous? what's "driver executor peakMemoryMetrics"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862023314
Thanks for the update @AngersZhuuuu, I am fine with merging this ... will keep it around for a couple of days in case there are other comments.
+CC @zhouyejoe, @thejdeep PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-809585304
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41235/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31992: [SPARK-34898][CORE] We should send SparkListenerExecutorMetricsUpdateEventLog of `driver` appropriately
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-814569934
> As I understood based on description and proposed changes, the right level of integration for this would be with metrics subsystem.. While we can do it via other means, it is not optimal to do so.
> Having said that, I dont have strong opinions on this ... will let some more involved with this comment better.
Yea, also ping @srowen @ngbinh @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31992: [SPARK-34898][CORE] We should log SparkListenerExecutorMetricsUpdateEvent of `driver` appropriately when `spark.eventLog.logStageExecutorMetrics` is true
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31992:
URL: https://github.com/apache/spark/pull/31992#issuecomment-862141748
**[Test build #139842 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139842/testReport)** for PR 31992 at commit [`6c81e2d`](https://github.com/apache/spark/commit/6c81e2dd74b7f49b3ba30b8618d1a502db1246dc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org