You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/28 10:10:18 UTC
[GitHub] [spark] sarutak opened a new pull request #32381: [SPARK-35229][SQL] Limit the maximum number of items on the timeline view.
sarutak opened a new pull request #32381:
URL: https://github.com/apache/spark/pull/32381
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR proposes to introduces three new configurations to limit the maximum number of jobs/stages/executors on the timeline view.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
If the number of items on the timeline view grows +1000, rendering can be significantly slow.
https://issues.apache.org/jira/browse/SPARK-35229
The maximum number of tasks on the timeline is already limited by `spark.ui.timeline.tasks.maximum` so l proposed to mitigate this issue with the same manner.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. the maximum number of items shown on the timeline view is limited.
I proposed the default value 500 for jobs and stages, and 250 for executors.
A executor has at most 2 items (added and removed) 250 is chosen.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
I manually confirm this change works with the following procedures.
```
# launch a cluster
$ bin/spark-shell --conf spark.ui.retainedDeadExecutors=300 --master "local-cluster[4, 1, 1024]"
// Confirm the maximum number of jobs
(1 to 1000).foreach { _ => sc.parallelize(List(1)).collect }
// Confirm the maximum number of stages
var df = sc.parallelize(1 to 2)
(1 to 1000).foreach { i => df = df.repartition(i % 5 + 1) }
df.collect
// Confirm the maximum number of executors
(1 to 300).foreach { _ => try sc.parallelize(List(1)).foreach { _ => System.exit(0) } catch { case e => }}
```
Screenshots here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834371135
**[Test build #138253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138253/testReport)** for PR 32381 at commit [`b7fa86a`](https://github.com/apache/spark/commit/b7fa86acae03eb3de01b05574052378076a0d632).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-828458068
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138042/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r627534327
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -192,6 +201,32 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</a>
</span> ++
<div id="application-timeline" class="collapsed">
+ {
+ if (MAX_TIMELINE_JOBS < jobs.size) {
+ <div>
+ <strong>
+ This page has more than the maximum number of jobs that can be shown in the
+ visualization! Only the most recent {MAX_TIMELINE_JOBS} submitted/completed jobs
+ (of {jobs.size} total) are shown.
Review comment:
Nit: how about just
```
Only the most recent {MAX_TIMELINE_JOBS} submitted jobs(of {jobs.size} total) are shown.
```
The sentence
```
This page has more than the maximum number of jobs that can be shown in the ...
```
seems redundant.
Not sure whether we should hint the user to tune the configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834371135
**[Test build #138253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138253/testReport)** for PR 32381 at commit [`b7fa86a`](https://github.com/apache/spark/commit/b7fa86acae03eb3de01b05574052378076a0d632).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #32381:
URL: https://github.com/apache/spark/pull/32381
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-828391255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834572080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r630114965
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala
##########
@@ -30,6 +30,7 @@ private[ui] class JobsTab(parent: SparkUI, store: AppStatusStore)
extends SparkUITab(parent, "jobs") {
val sc = parent.sc
+ val conf = parent.conf
Review comment:
Yes. Without this, we can't access `SparkConf` from `AllJobsPage` and `JobPage`.
We can't do `parent.parent.conf` from those pages.
`StageTab` has the same code for `StagePage`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r630083673
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala
##########
@@ -30,6 +30,7 @@ private[ui] class JobsTab(parent: SparkUI, store: AppStatusStore)
extends SparkUITab(parent, "jobs") {
val sc = parent.sc
+ val conf = parent.conf
Review comment:
is this needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r627530249
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -64,9 +68,12 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</svg></div>.toString.filter(_ != '\n')
private def makeJobEvent(jobs: Seq[v1.JobData]): Seq[String] = {
+ val now = System.currentTimeMillis()
jobs.filter { job =>
job.status != JobExecutionStatus.UNKNOWN && job.submissionTime.isDefined
- }.map { job =>
+ }.sortBy { j =>
+ (-j.completionTime.map(_.getTime).getOrElse(now), -j.submissionTime.get.getTime)
+ }.take(MAX_TIMELINE_JOBS).map { job =>
Review comment:
nit: how about
sortBy(completionTime, submissionTime).takeRight
the negative value is not straightforward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r627115998
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -192,6 +201,32 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</a>
</span> ++
<div id="application-timeline" class="collapsed">
+ {
+ if (MAX_TIMELINE_JOBS < jobs.size) {
+ <div>
+ <strong>
+ This page has more than the maximum number of jobs that can be shown in the
Review comment:
I think we should focus on resolving this issue itself for now.
The timeline view for StagePage shows similar message so if it's necessary, let's change the color for all the warning message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-833270636
cc @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r628198588
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -192,6 +201,32 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</a>
</span> ++
<div id="application-timeline" class="collapsed">
+ {
+ if (MAX_TIMELINE_JOBS < jobs.size) {
+ <div>
+ <strong>
+ This page has more than the maximum number of jobs that can be shown in the
+ visualization! Only the most recent {MAX_TIMELINE_JOBS} submitted/completed jobs
+ (of {jobs.size} total) are shown.
Review comment:
Thanks. This sentence is brought from `StagePage.scala` so I've modified it too.
As I added descriptions for the new configurations, I don't add hint in the sentence. If there are any feedback from users, let's reconsider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830646657
**[Test build #138130 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138130/testReport)** for PR 32381 at commit [`a7cb930`](https://github.com/apache/spark/commit/a7cb9307a520f663e4cbb2a3baff524f260d0ef1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-833274643
@sarutak I am back! I will review this one today :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SaurabhChawla100 commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r624878008
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -192,6 +201,32 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</a>
</span> ++
<div id="application-timeline" class="collapsed">
+ {
+ if (MAX_TIMELINE_JOBS < jobs.size) {
+ <div>
+ <strong>
+ This page has more than the maximum number of jobs that can be shown in the
Review comment:
Just thinking -> Shall we have this alert in red colour since this is one of the scenario where we are showing the limited information
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -118,7 +125,9 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
private def makeExecutorEvent(executors: Seq[v1.ExecutorSummary]):
Seq[String] = {
val events = ListBuffer[String]()
- executors.foreach { e =>
+ executors.sortBy { e =>
+ -math.max(e.addTime.getTime, e.removeTime.map(_.getTime).getOrElse(-1L))
Review comment:
shall we do it in the same way by sorting with both the attributes as done in case of jobs and stages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-838420456
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r627121264
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -118,7 +125,9 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
private def makeExecutorEvent(executors: Seq[v1.ExecutorSummary]):
Seq[String] = {
val events = ListBuffer[String]()
- executors.foreach { e =>
+ executors.sortBy { e =>
+ -math.max(e.addTime.getTime, e.removeTime.map(_.getTime).getOrElse(-1L))
Review comment:
I don't think it's good for executors. If we do so, removed executor can disappear from the view even if the removal event is the most latest event.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r627534327
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -192,6 +201,32 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</a>
</span> ++
<div id="application-timeline" class="collapsed">
+ {
+ if (MAX_TIMELINE_JOBS < jobs.size) {
+ <div>
+ <strong>
+ This page has more than the maximum number of jobs that can be shown in the
+ visualization! Only the most recent {MAX_TIMELINE_JOBS} submitted/completed jobs
+ (of {jobs.size} total) are shown.
Review comment:
Nit: how about just
```
Only the most recent {MAX_TIMELINE_JOBS} submitted jobs(of {jobs.size} total) are shown.
```
Not sure about whether we should hint the user to tune the configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834426505
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42775/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-828391255
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42561/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830626989
**[Test build #138130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138130/testReport)** for PR 32381 at commit [`a7cb930`](https://github.com/apache/spark/commit/a7cb9307a520f663e4cbb2a3baff524f260d0ef1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-833270865
Oh... He is in vacation...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r628198588
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -192,6 +201,32 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</a>
</span> ++
<div id="application-timeline" class="collapsed">
+ {
+ if (MAX_TIMELINE_JOBS < jobs.size) {
+ <div>
+ <strong>
+ This page has more than the maximum number of jobs that can be shown in the
+ visualization! Only the most recent {MAX_TIMELINE_JOBS} submitted/completed jobs
+ (of {jobs.size} total) are shown.
Review comment:
Thanks. This sentence is brought from `StagePage.scala` so I've modified it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r623301421
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -66,7 +70,9 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
private def makeJobEvent(jobs: Seq[v1.JobData]): Seq[String] = {
jobs.filter { job =>
job.status != JobExecutionStatus.UNKNOWN && job.submissionTime.isDefined
- }.map { job =>
+ }.sortBy { j =>
+ -math.max(j.submissionTime.get.getTime, j.completionTime.map(_.getTime).getOrElse(-1L))
Review comment:
I reconsidered that we should choose `System.currentTimeMillis` rather than `-1` because it's a in progress job/stage.
O.K, let's sort by both start and end (I have no time today so I'll do it tommorrow).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SaurabhChawla100 commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r622787382
##########
File path: core/src/main/scala/org/apache/spark/internal/config/UI.scala
##########
@@ -129,6 +129,21 @@ private[spark] object UI {
.intConf
.createWithDefault(1000)
+ val UI_TIMELINE_JOBS_MAXIMUM = ConfigBuilder("spark.ui.timeline.jobs.maximum")
Review comment:
Instead of introducing the new config can we try using these config
`val MAX_RETAINED_JOBS = ConfigBuilder("spark.ui.retainedJobs") .version("1.2.0") .intConf .createWithDefault(1000)`
same is for the stages(spark.ui.retainedStages)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830635504
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r627538288
##########
File path: core/src/main/scala/org/apache/spark/internal/config/UI.scala
##########
@@ -129,6 +129,21 @@ private[spark] object UI {
.intConf
.createWithDefault(1000)
+ val UI_TIMELINE_JOBS_MAXIMUM = ConfigBuilder("spark.ui.timeline.jobs.maximum")
Review comment:
Shall we write down these new configurations into `configuration.md`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834426505
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42775/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830635504
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834423647
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42775/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830626989
**[Test build #138130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138130/testReport)** for PR 32381 at commit [`a7cb930`](https://github.com/apache/spark/commit/a7cb9307a520f663e4cbb2a3baff524f260d0ef1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r623140875
##########
File path: core/src/main/scala/org/apache/spark/internal/config/UI.scala
##########
@@ -129,6 +129,21 @@ private[spark] object UI {
.intConf
.createWithDefault(1000)
+ val UI_TIMELINE_JOBS_MAXIMUM = ConfigBuilder("spark.ui.timeline.jobs.maximum")
Review comment:
How many records for jobs information that user want them to be retained is not always the same as how many items displayed in the timeline view.
Users can want much more records to be retained than the default number but want less items to be shown.
So I prefer to have a new property for jobs. The same applies to stages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-828348579
**[Test build #138042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138042/testReport)** for PR 32381 at commit [`43b2938`](https://github.com/apache/spark/commit/43b293865b5346fb66889ed8900e5fca297b1732).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-828391195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830652137
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138130/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834572080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r628197847
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -64,9 +68,12 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
</svg></div>.toString.filter(_ != '\n')
private def makeJobEvent(jobs: Seq[v1.JobData]): Seq[String] = {
+ val now = System.currentTimeMillis()
jobs.filter { job =>
job.status != JobExecutionStatus.UNKNOWN && job.submissionTime.isDefined
- }.map { job =>
+ }.sortBy { j =>
+ (-j.completionTime.map(_.getTime).getOrElse(now), -j.submissionTime.get.getTime)
+ }.take(MAX_TIMELINE_JOBS).map { job =>
Review comment:
Thanks. It seems simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-828348579
**[Test build #138042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138042/testReport)** for PR 32381 at commit [`43b2938`](https://github.com/apache/spark/commit/43b293865b5346fb66889ed8900e5fca297b1732).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-837683190
+CC @zhouyejoe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830633105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-828443444
**[Test build #138042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138042/testReport)** for PR 32381 at commit [`43b2938`](https://github.com/apache/spark/commit/43b293865b5346fb66889ed8900e5fca297b1732).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SaurabhChawla100 commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SaurabhChawla100 commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r623284990
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -66,7 +70,9 @@ private[ui] class AllJobsPage(parent: JobsTab, store: AppStatusStore) extends We
private def makeJobEvent(jobs: Seq[v1.JobData]): Seq[String] = {
jobs.filter { job =>
job.status != JobExecutionStatus.UNKNOWN && job.submissionTime.isDefined
- }.map { job =>
+ }.sortBy { j =>
+ -math.max(j.submissionTime.get.getTime, j.completionTime.map(_.getTime).getOrElse(-1L))
Review comment:
We are here considering the latest submitted or latest completed jobs based on the submissionTime and completionTime
So If we take this example
```
scala> case class Test(startTime: Long, endTime: Long)
defined class Test
val addSeq: Seq[Test] = Seq(Test(2,7), Test(1,5), Test(4,5), Test(2,5), Test(3,4), Test(1,-1))
scala> addSeq
res135: Seq[Test] = List(Test(1,5), Test(2,7), Test(3,4), Test(4,5), Test(2,5), Test(1,-1))
```
If we take latest startTime and latest endTime and take only 2 items from the seq than it would be
List(Test(2,7), Test(4,5))
where as per the above code by taking the max for startTime and endTime
```
scala> addSeq.sortBy { e => -math.max( e.startTime, e.endTime) }
res136: Seq[Test] = List(Test(2,7), Test(1,5), Test(4,5), Test(2,5), Test(3,4), Test(1,-1))
scala> addSeq.sortBy { e => -math.max( e.startTime, e.endTime) }.take(2)
res137: Seq[Test] = List(Test(2,7), Test(1,5))
```
if we take both the tuples and compare the result we can get the latest start and latest end value
```
scala> addSeq.sortWith{ (t1,t2) => !(t1.startTime < t2.startTime || t1.endTime < t2.endTime)}
res138: Seq[Test] = List(Test(2,7), Test(4,5), Test(2,5), Test(1,5), Test(3,4), Test(1,-1))
scala> addSeq.sortWith{ (t1,t2) => !(t1.startTime < t2.startTime || t1.endTime < t2.endTime)}.take(2)
res139: Seq[Test] = List(Test(2,7), Test(4,5))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r628200275
##########
File path: core/src/main/scala/org/apache/spark/internal/config/UI.scala
##########
@@ -129,6 +129,21 @@ private[spark] object UI {
.intConf
.createWithDefault(1000)
+ val UI_TIMELINE_JOBS_MAXIMUM = ConfigBuilder("spark.ui.timeline.jobs.maximum")
Review comment:
I've added descriptions about the new properties and the existing `spark.ui.timeline.tasks.maximum`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-830652137
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138130/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32381: [SPARK-35229][WEBUI] Limit the maximum number of items on the timeline view.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32381:
URL: https://github.com/apache/spark/pull/32381#issuecomment-834559721
**[Test build #138253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138253/testReport)** for PR 32381 at commit [`b7fa86a`](https://github.com/apache/spark/commit/b7fa86acae03eb3de01b05574052378076a0d632).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org