You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/18 14:20:18 UTC
[GitHub] [spark] zhli1142015 opened a new pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
zhli1142015 opened a new pull request #28859:
URL: https://github.com/apache/spark/pull/28859
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR is for fixing bug [32024](https://issues.apache.org/jira/browse/SPARK-32024). We found after history server restart, we would randomly see error "java.lang.IllegalStateException: Disk usage tracker went negative (now = -***, delta = -***)" from `HistoryServerDiskManager`.
**Cause**: Reading level db would trigger table file compaction, then it may cause size of level db directory changed. This size change may not be captured by `ApplicationStoreInfo`. When service restarts, `currentUsage` is calculated from real directory size, this value may be less then sum of `ApplicationStoreInfo.size`. In `makeRoom()` function, outdated value of `ApplicationStoreInfo.size` is used to update usage. Then `currentUsage` becomes negative.
**Reproduce**: we can reduce config value of "spark.history.retainedApplications" and "spark.history.store.maxDiskUsage" to reproduce this issue in dev environment. 1. start history server, load some applications and access some pages (maybe "stages" page trigger leveldb compaction). 2. restart HS, and refresh pages.
I also write an UT to simulate this case.
**Benefit**: this change would help improve history server reliability.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Add unit test and manually tested it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655539393
Honestly, I do merge too time to time e.g.) when there are some reasons or when the Jenkins build itself is completely orthogonal such as fixing AppVeyor CI issue .. but should be best to stick to the standard in general :-).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652743648
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r443225227
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
##########
@@ -83,6 +83,20 @@ private class HistoryServerDiskManager(
listing.delete(info.getClass(), info.path)
}
+ // Reading level db would trigger table file compaction, then it may cause size of level db
Review comment:
Thanks for your comments, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652963767
**[Test build #124897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124897/testReport)** for PR 28859 at commit [`b9057ec`](https://github.com/apache/spark/commit/b9057ecaa575e7f09c74bd9f52c0d3cc65e24bb9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654645126
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125173/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655751281
Thanks @HeartSaVioR and @dongjoon-hyun . i can reproduce it on branch 3.0 also, the cause is same as @HeartSaVioR mentioned, commit 245aee9 is not in branch 2.4 and 3.0. how about fix test case "SPARK-32024: update ApplicationStoreInfo.size during initializing" in default , 2.4 and 3.0, let it only covers this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653932725
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124947/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655191991
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655703992
If `branch-3.0` is broken, `master` branch also looks suspicious.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655422666
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655194334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653946282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-646937829
cc @gengliangwang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652741848
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r449338597
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
##########
@@ -75,14 +75,32 @@ private class HistoryServerDiskManager(
// Go through the recorded store directories and remove any that may have been removed by
// external code.
- val orphans = listing.view(classOf[ApplicationStoreInfo]).asScala.filter { info =>
- !new File(info.path).exists()
- }.toSeq
+ val (existing, orphans) = listing
Review comment:
nit: existing => existences
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655330476
**[Test build #125313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125313/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654093219
**[Test build #125054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125054/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652743318
**[Test build #124847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124847/testReport)** for PR 28859 at commit [`b9057ec`](https://github.com/apache/spark/commit/b9057ecaa575e7f09c74bd9f52c0d3cc65e24bb9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652960456
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654200677
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654323566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655752902
I'll submit a hotfix patch against master (can cherry-pick to broken branches) once I confirmed my fix works for branch-2.4 as well (it should work with master as well, as it "adds" the when statement).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654322112
**[Test build #125079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125079/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655327813
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654441441
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652964952
**[Test build #124897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124897/testReport)** for PR 28859 at commit [`b9057ec`](https://github.com/apache/spark/commit/b9057ecaa575e7f09c74bd9f52c0d3cc65e24bb9).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652744590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652964985
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654048649
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652744590
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655090327
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655090327
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654584903
**[Test build #125173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125173/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654439910
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r443216190
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,26 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test ("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
Review comment:
nit: no space between `test` and `(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654644083
**[Test build #125173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125173/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654323578
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125079/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655744956
Thanks for sharing the test failure, @dongjoon-hyun . I took a quick look, and realized 245aee9fc85c22352bb946a8987229270ba6069f was landed only on master branch (as it's an improvement) which got rid of size call in destination path, whereas branch-3.0 and branch-2.4 don't.
Below is the commit I fixed the test in branch-3.0 (confirmed test passed):
https://github.com/HeartSaVioR/spark/commit/6a36c9c072d0c7aeb29e12439d66d8205e2d5d7d
and I also confirmed the test pass if we cherry pick 245aee9fc85c22352bb946a8987229270ba6069f to branch-3.0:
https://github.com/HeartSaVioR/spark/commit/c8c9531858aa1d59fdb948d8840ea1f0ad2f1415
I'm now applying the fix to branch-2.4 - IDE requires rebuild on switching 3.x vs 2.x so it'd take time, but the mechanism of the fix should be applied to 2.4 as well.
Would we like to go with hotfix, or cherry pick 245aee9fc85c22352bb946a8987229270ba6069f?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655257055
**[Test build #125267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125267/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654455958
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125124/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655327813
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653932717
**[Test build #124947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124947/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655744956
Thanks for sharing the test failure, @dongjoon-hyun . I took a quick look, and realized 245aee9fc85c22352bb946a8987229270ba6069f was landed only on master branch (as it's an improvement) which got rid of size call in destination path, whereas branch-3.0 and branch-2.4 don't.
Below is the commit I fixed the test in branch-3.0:
https://github.com/HeartSaVioR/spark/commit/6a36c9c072d0c7aeb29e12439d66d8205e2d5d7d
and I also confirmed the test pass if we cherry pick 245aee9fc85c22352bb946a8987229270ba6069f to branch-3.0:
https://github.com/HeartSaVioR/spark/commit/c8c9531858aa1d59fdb948d8840ea1f0ad2f1415
I'm now applying the fix to branch-2.4 - IDE requires rebuild on switching 3.x vs 2.x so it'd take time, but the mechanism of the fix should be applied to 2.4 as well.
Would we like to go with hotfix, or cherry pick 245aee9fc85c22352bb946a8987229270ba6069f?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654192090
**[Test build #125054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125054/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654194495
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655190088
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-646050511
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r443225264
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,26 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test ("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
Review comment:
thanks, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655504143
Thanks @zhli1142015 for the contribution! Merged into master, branch-3.0, branch-2.4 respectively.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652743648
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655745634
Probably better to land the hotfix into master as well if we go with hotfix, as it looks to be safer assumption rather than depending on 245aee9.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r443215446
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
##########
@@ -83,6 +83,20 @@ private class HistoryServerDiskManager(
listing.delete(info.getClass(), info.path)
}
+ // Reading level db would trigger table file compaction, then it may cause size of level db
Review comment:
We could apply "partition" instead of filter in L78, so that we only call iterate with view once and retrieve two lists (orphans, existing) - L90 can use existing to avoid calling view again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654202458
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652959861
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655422675
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125313/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654455948
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655325185
The PR builders look to be quite unstable, though all the test failures are not related to the change.
> build 125054
org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> build 125079
org.apache.spark.sql.hive.thriftserver.CliSuite.xml
> build 125250
org.apache.spark.sql.kafka010.KafkaRelationSuiteV1.timestamp provided for starting, ending not provided
org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> build 125267
org.apache.spark.sql.execution.DataSourceV2ScanExecRedactionSuite.SPARK-30362: test input metrics for DSV2
org.apache.spark.sql.hive.thriftserver.CliSuite.Pad Decimal numbers with trailing zeros to the scale of the column
There's no consistent failure on specific test in these builds, which represents the change doesn't break the test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655501284
Thank you for your helping @HeartSaVioR . cc @gengliangwang , is it ok to merge this PR, or I need to run test again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652744578
**[Test build #124847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124847/testReport)** for PR 28859 at commit [`b9057ec`](https://github.com/apache/spark/commit/b9057ecaa575e7f09c74bd9f52c0d3cc65e24bb9).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654200570
```
org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)
Error Details
org.mockito.exceptions.base.MockitoException: ClassCastException occurred while creating the mockito mock : class to mock : 'org.apache.hive.service.cli.session.SessionManager', loaded by classloader : 'sun.misc.Launcher$AppClassLoader@2d6d8735' created class : 'org.mockito.codegen.SessionManager$MockitoMock$930162926', loaded by classloader : 'net.bytebuddy.dynamic.loading.MultipleParentClassLoader@11e68869' proxy instance class : 'org.mockito.codegen.SessionManager$MockitoMock$930162926', loaded by classloader : 'net.bytebuddy.dynamic.loading.MultipleParentClassLoader@11e68869' instance creation by : ObjenesisInstantiator You might experience classloading issues, please ask the mockito mailing-list.
Stack Trace
sbt.ForkMain$ForkError: org.mockito.exceptions.base.MockitoException:
ClassCastException occurred while creating the mockito mock :
class to mock : 'org.apache.hive.service.cli.session.SessionManager', loaded by classloader : 'sun.misc.Launcher$AppClassLoader@2d6d8735'
created class : 'org.mockito.codegen.SessionManager$MockitoMock$930162926', loaded by classloader : 'net.bytebuddy.dynamic.loading.MultipleParentClassLoader@11e68869'
proxy instance class : 'org.mockito.codegen.SessionManager$MockitoMock$930162926', loaded by classloader : 'net.bytebuddy.dynamic.loading.MultipleParentClassLoader@11e68869'
instance creation by : ObjenesisInstantiator
You might experience classloading issues, please ask the mockito mailing-list.
at org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.beforeAll(HiveSessionImplSuite.scala:44)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:59)
at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:317)
at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:510)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: sbt.ForkMain$ForkError: java.lang.ClassCastException: org.mockito.codegen.SessionManager$MockitoMock$930162926 cannot be cast to org.mockito.internal.creation.bytebuddy.MockAccess
at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:48)
at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
at org.mockito.internal.MockitoCore.mock(MockitoCore.java:63)
at org.mockito.Mockito.mock(Mockito.java:1908)
at org.mockito.Mockito.mock(Mockito.java:1817)
... 13 more
```
This seems to be seen quite frequently. Probably worth to file an issue for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r443216605
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,26 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test ("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app2", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
+ // Simulate: service restarts, new disk manager (manager1) is initialized.
+ val manager1 = mockManager()
+ // Simulate: leveldb compaction before restart, directory size reduces.
+ doReturn(2L).when(manager1).sizeOf(meq(dstA))
+ doReturn(2L).when(manager1).sizeOf(meq(new File(testDir, "apps")))
+ manager1.initialize()
+ assert(manager1.free() === 1)
+ // If "ApplicationStoreInfo.size" is not correctly updated, "IllegalStateException"
+ // would be thrown.
+ val leaseB = manager1.lease(2)
+ assert(manager1.free() === 1)
+ leaseB.rollback()
+ assert(manager1.free() === 3)
Review comment:
Probably better to simulate another case: actual directory size is bigger than appinfo - some cache entities can be written after replaying, which "might" affect the size.
(That said, the mismatch between appinfo and actual dir size also occurs during run, though it may not matter unless there's no reflection.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654583873
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655190097
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125250/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-646050511
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655522336
Hey @HeartSaVioR, let's don't merge PRs when tests are being failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r444811210
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,35 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app1", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
+ // Simulate: service restarts, new disk manager (manager1) is initialized.
+ val manager1 = mockManager()
+ // Simulate: leveldb compaction before restart, directory size reduces.
+ doReturn(2L).when(manager1).sizeOf(meq(dstA))
+ doReturn(2L).when(manager1).sizeOf(meq(new File(testDir, "apps")))
+ manager1.initialize()
+ assert(manager1.free() === 1)
+ // If "ApplicationStoreInfo.size" is not correctly updated, "IllegalStateException"
+ // would be thrown.
+ val leaseB = manager1.lease(2)
+ assert(manager1.free() === 1)
+ doReturn(2L).when(manager1).sizeOf(meq(leaseB.tmpPath))
+ val dstB = leaseB.commit("app2", None)
+ assert(manager1.committed() === 2)
+ val manager2 = mockManager()
+ // Simulate: cache entities are written after replaying, directory size increases.
+ doReturn(3L).when(manager2).sizeOf(meq(dstB))
+ doReturn(3L).when(manager2).sizeOf(meq(new File(testDir, "apps")))
+ manager2.initialize()
+ assert(manager2.free() === 0)
+ val leaseC = manager2.lease(2)
Review comment:
Better to explicitly mention it leads eviction on cached entities, hence free() goes to 1.
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,35 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app1", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
Review comment:
Could we track and verify ApplicationStoreInfo in store as well? I see the test represents how HistoryServerDiskManager fails without the patch, but it would be pretty much intuitive if we also verify what we've changed directly.
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,35 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app1", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
+ // Simulate: service restarts, new disk manager (manager1) is initialized.
+ val manager1 = mockManager()
+ // Simulate: leveldb compaction before restart, directory size reduces.
+ doReturn(2L).when(manager1).sizeOf(meq(dstA))
+ doReturn(2L).when(manager1).sizeOf(meq(new File(testDir, "apps")))
+ manager1.initialize()
+ assert(manager1.free() === 1)
+ // If "ApplicationStoreInfo.size" is not correctly updated, "IllegalStateException"
+ // would be thrown.
+ val leaseB = manager1.lease(2)
+ assert(manager1.free() === 1)
+ doReturn(2L).when(manager1).sizeOf(meq(leaseB.tmpPath))
+ val dstB = leaseB.commit("app2", None)
+ assert(manager1.committed() === 2)
+ val manager2 = mockManager()
Review comment:
Let's add some empty lines in overall to bring readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655194334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652215562
Gently ping, @jiangxb1987 , @dongjoon-hyun , @ueshin , @gengliangwang , could you please help to review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655422666
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654506797
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653273688
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653932723
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653932391
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653946282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655420389
**[Test build #125313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125313/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652964991
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124897/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653946336
**[Test build #124969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124969/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654440861
**[Test build #125124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125124/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654048653
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124969/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r444840208
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,35 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app1", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
Review comment:
Good idea, updated.
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,35 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app1", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
+ // Simulate: service restarts, new disk manager (manager1) is initialized.
+ val manager1 = mockManager()
+ // Simulate: leveldb compaction before restart, directory size reduces.
+ doReturn(2L).when(manager1).sizeOf(meq(dstA))
+ doReturn(2L).when(manager1).sizeOf(meq(new File(testDir, "apps")))
+ manager1.initialize()
+ assert(manager1.free() === 1)
+ // If "ApplicationStoreInfo.size" is not correctly updated, "IllegalStateException"
+ // would be thrown.
+ val leaseB = manager1.lease(2)
+ assert(manager1.free() === 1)
+ doReturn(2L).when(manager1).sizeOf(meq(leaseB.tmpPath))
+ val dstB = leaseB.commit("app2", None)
+ assert(manager1.committed() === 2)
+ val manager2 = mockManager()
+ // Simulate: cache entities are written after replaying, directory size increases.
+ doReturn(3L).when(manager2).sizeOf(meq(dstB))
+ doReturn(3L).when(manager2).sizeOf(meq(new File(testDir, "apps")))
+ manager2.initialize()
+ assert(manager2.free() === 0)
+ val leaseC = manager2.lease(2)
Review comment:
Thanks, updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654091513
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654093800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654455877
**[Test build #125124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125124/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653946129
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-646051182
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653932391
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653932723
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-646051182
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r449350011
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
##########
@@ -75,14 +75,32 @@ private class HistoryServerDiskManager(
// Go through the recorded store directories and remove any that may have been removed by
// external code.
- val orphans = listing.view(classOf[ApplicationStoreInfo]).asScala.filter { info =>
- !new File(info.path).exists()
- }.toSeq
+ val (existing, orphans) = listing
+ .view(classOf[ApplicationStoreInfo])
+ .asScala
+ .toSeq
+ .partition { info =>
+ new File(info.path).exists()
+ }
orphans.foreach { info =>
listing.delete(info.getClass(), info.path)
}
+ // Reading level db would trigger table file compaction, then it may cause size of level db
+ // directory changed. When service restarts, "currentUsage" is calculated from real directory
+ // size. Update "ApplicationStoreInfo.size" to ensure "currentUsage" equals
+ // sum of "ApplicationStoreInfo.size".
+ val changedStoreInfo = existing.filter { info =>
Review comment:
thanks, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r444840003
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,35 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app1", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
+ // Simulate: service restarts, new disk manager (manager1) is initialized.
+ val manager1 = mockManager()
+ // Simulate: leveldb compaction before restart, directory size reduces.
+ doReturn(2L).when(manager1).sizeOf(meq(dstA))
+ doReturn(2L).when(manager1).sizeOf(meq(new File(testDir, "apps")))
+ manager1.initialize()
+ assert(manager1.free() === 1)
+ // If "ApplicationStoreInfo.size" is not correctly updated, "IllegalStateException"
+ // would be thrown.
+ val leaseB = manager1.lease(2)
+ assert(manager1.free() === 1)
+ doReturn(2L).when(manager1).sizeOf(meq(leaseB.tmpPath))
+ val dstB = leaseB.commit("app2", None)
+ assert(manager1.committed() === 2)
+ val manager2 = mockManager()
Review comment:
Thanks, updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654455948
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655490854
> build 125313
org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654585030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655530835
Oh OK. Thanks for noticing. I hope we'd be a bit flexible and pragmatic, but I also sincerely respect the rule and policy. I'll follow it in next time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654440861
**[Test build #125124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125124/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655088482
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654323566
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654584903
**[Test build #125173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125173/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652963767
**[Test build #124897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124897/testReport)** for PR 28859 at commit [`b9057ec`](https://github.com/apache/spark/commit/b9057ecaa575e7f09c74bd9f52c0d3cc65e24bb9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654093800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655703992
If `branch-3.0` is broken, `master` branch also looks suspicious.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654201970
**[Test build #125079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125079/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655634933
Could you recover this please via a follow-up or a revert, @HeartSaVioR ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655090338
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29861/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655702826
It turns out this breaks `branch-3.0`, too
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-maven-hadoop-2.7-hive-1.2/lastCompletedBuild/testReport/org.apache.spark.deploy.history/HistoryServerDiskManagerSuite/SPARK_32024__update_ApplicationStoreInfo_size_during_initializing/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654201970
**[Test build #125079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125079/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654202458
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655257914
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125267/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655089669
**[Test build #125250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125250/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655327071
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655193968
**[Test build #125267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125267/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655089669
**[Test build #125250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125250/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #28859:
URL: https://github.com/apache/spark/pull/28859
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655768498
Thank you, @HeartSaVioR !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654441441
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654093219
**[Test build #125054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125054/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655190088
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655189423
**[Test build #125250 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125250/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653931981
**[Test build #124947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124947/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655257909
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654048649
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655193968
**[Test build #125267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125267/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652744597
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124847/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654645120
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654645120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655633880
Hi, All. This breaks `branch-2.4` UT. I didn't take a look at the other branches.
```
- SPARK-32024: update ApplicationStoreInfo.size during initializing *** FAILED ***
0 did not equal 3 (HistoryServerDiskManagerSuite.scala:168)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655327021
Let me trigger the test once more. If that also fails with flaky test we'd probably be better to just merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654194495
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-646054069
@jiangxb1987 , @dongjoon-hyun , @ueshin , Could you please help to review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655257909
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653931981
**[Test build #124947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124947/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-648049709
Gently ping, @HeartSaVioR , could you please help review again?
@jiangxb1987 , @dongjoon-hyun , @ueshin , @gengliangwang , could you please help to review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-653258728
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652964985
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655330476
**[Test build #125313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125313/testReport)** for PR 28859 at commit [`ed05872`](https://github.com/apache/spark/commit/ed05872a5b6fd8ceb38b93636b1c3c93cc70288e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r449339030
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
##########
@@ -75,14 +75,32 @@ private class HistoryServerDiskManager(
// Go through the recorded store directories and remove any that may have been removed by
// external code.
- val orphans = listing.view(classOf[ApplicationStoreInfo]).asScala.filter { info =>
- !new File(info.path).exists()
- }.toSeq
+ val (existing, orphans) = listing
+ .view(classOf[ApplicationStoreInfo])
+ .asScala
+ .toSeq
+ .partition { info =>
+ new File(info.path).exists()
+ }
orphans.foreach { info =>
listing.delete(info.getClass(), info.path)
}
+ // Reading level db would trigger table file compaction, then it may cause size of level db
+ // directory changed. When service restarts, "currentUsage" is calculated from real directory
+ // size. Update "ApplicationStoreInfo.size" to ensure "currentUsage" equals
+ // sum of "ApplicationStoreInfo.size".
+ val changedStoreInfo = existing.filter { info =>
Review comment:
Nit: I think we can write this in one loop:
```
existing.foreach { info =>
val fileSize = sizeOf(new File(info.path)
if (fileSize != info.size) {
listing.write(info.copy(size = fileSize))
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r443225314
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerDiskManagerSuite.scala
##########
@@ -158,4 +158,26 @@ class HistoryServerDiskManagerSuite extends SparkFunSuite with BeforeAndAfter {
assert(manager.approximateSize(50L, true) > 50L)
}
+ test ("SPARK-32024: update ApplicationStoreInfo.size during initializing") {
+ val manager = mockManager()
+ val leaseA = manager.lease(2)
+ doReturn(3L).when(manager).sizeOf(meq(leaseA.tmpPath))
+ val dstA = leaseA.commit("app2", None)
+ assert(manager.free() === 0)
+ assert(manager.committed() === 3)
+ // Simulate: service restarts, new disk manager (manager1) is initialized.
+ val manager1 = mockManager()
+ // Simulate: leveldb compaction before restart, directory size reduces.
+ doReturn(2L).when(manager1).sizeOf(meq(dstA))
+ doReturn(2L).when(manager1).sizeOf(meq(new File(testDir, "apps")))
+ manager1.initialize()
+ assert(manager1.free() === 1)
+ // If "ApplicationStoreInfo.size" is not correctly updated, "IllegalStateException"
+ // would be thrown.
+ val leaseB = manager1.lease(2)
+ assert(manager1.free() === 1)
+ leaseB.rollback()
+ assert(manager1.free() === 3)
Review comment:
Thanks, this is a valid case, i added it in my UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654194505
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125054/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-654585030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655773333
OK confirmed the hotfix works in master and branch-2.4 as well. Will submit a hotfix soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652960456
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-655325185
The PR builders look to be quite unstable, though all the test failures are not related to the change.
> build 125054
org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> build 125079
org.apache.spark.sql.hive.thriftserver.CliSuite.xml
> build 125250
org.apache.spark.sql.kafka010.KafkaRelationSuiteV1.timestamp provided for starting, ending not provided
org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> build 125267
org.apache.spark.sql.execution.DataSourceV2ScanExecRedactionSuite.SPARK-30362: test input metrics for DSV2
org.apache.spark.sql.hive.thriftserver.CliSuite.Pad Decimal numbers with trailing zeros to the scale of the column
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28859:
URL: https://github.com/apache/spark/pull/28859#issuecomment-652743318
**[Test build #124847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124847/testReport)** for PR 28859 at commit [`b9057ec`](https://github.com/apache/spark/commit/b9057ecaa575e7f09c74bd9f52c0d3cc65e24bb9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhli1142015 commented on a change in pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
Posted by GitBox <gi...@apache.org>.
zhli1142015 commented on a change in pull request #28859:
URL: https://github.com/apache/spark/pull/28859#discussion_r443225227
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala
##########
@@ -83,6 +83,20 @@ private class HistoryServerDiskManager(
listing.delete(info.getClass(), info.path)
}
+ // Reading level db would trigger table file compaction, then it may cause size of level db
Review comment:
@HeartSaVioR , thank you for your comments, updated it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org