You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/18 16:40:55 UTC
[GitHub] [spark] hai-tao-1 opened a new pull request, #36597: [SPARK-39225][CORE] History Server Update BatchSize
hai-tao-1 opened a new pull request, #36597:
URL: https://github.com/apache/spark/pull/36597
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
- Add a spark property, spark.history.fs.update.batchSize(default 1000), which specifies the batch size for updating new eventlog files.
- For each eventlog scan/parse process, only a max of `spark.history.fs.update.batchSize` jobs are scanned/parsed. This allows the scan process ends within a reasonable time, so new eventlog files(appeared after the scan process starts) can be scanned sooner.
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
Current Spark History Server suffers when there are a large number of eventlog files under eventLog.dir: when a SHS starts, the initial scan may take a long time, and new eventlog files would not be scanned/parsed until the initial scan completes.
For example, if the initial scan takes 1-2 days(this is not uncommon in large environments), the newly finished spark jobs would not show up in SHS since their eventlog files are not scanned/parsed until the initial scan process finishes. This would result in a 1-2 days SHS malfunctioning since the newly finished spark jobs are most likely to be queried by users.
This PR adds a limit for each scan process to `spark.history.fs.update.batchSize`, so that each scan can finish within a reasonable time(default 1000 jobs can usually be scanned in ~10-15 mins but this can vary depending on the sizes of the eventlog files). This will prevent the long initial scan process from blocking the newly appeared eventlog files be scanned.
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
No.
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
Tested locally:
- Tested with unit tests in the core module and all passed.
- Placed 10 eventlog files under `eventLog.dir`, and set `spark.history.fs.update.batchSize` to 2. When the SHS started, observed that the scan process parsed 2 files at each time(starting from the files with the latest modification time).
- Added a new eventlog file after the first scan started, and observed that the 2nd scan process picked up and parsed the new eventlog file before parsing other remaining (older) eventlog files.
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hai-tao-1 commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
hai-tao-1 commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876412532
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -494,6 +495,19 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
}
}
.flatMap { entry => EventLogFileReader(fs, entry) }
+ .filter { reader =>
+ try {
+ reader.modificationTime
+ true
+ } catch {
+ case e: Exception =>
Review Comment:
Changed to be more specific. Thanks.
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -494,6 +495,19 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
}
}
.flatMap { entry => EventLogFileReader(fs, entry) }
+ .filter { reader =>
+ try {
+ reader.modificationTime
+ true
+ } catch {
+ case e: Exception =>
Review Comment:
Changed to be more specific. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #36597: [SPARK-39225][CORE] Support `spark.history.fs.update.batchSize`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #36597:
URL: https://github.com/apache/spark/pull/36597#issuecomment-1131715634
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36597:
URL: https://github.com/apache/spark/pull/36597#issuecomment-1131291360
Thank you for updates, @hai-tao-1 . Yes, the only remaining comment is the test case.
> We need a test case for the configuration. Please check the corner cases especially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876456562
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -474,6 +474,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
* applications that haven't been updated since last time the logs were checked.
Review Comment:
Please add some description about the new logic here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hai-tao-1 commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
hai-tao-1 commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876509868
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -41,6 +41,15 @@ private[spark] object History {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
+ val UPDATE_BATCHSIZE = ConfigBuilder("spark.history.fs.update.batchSize")
+ .doc("Specifies the batch size for updating new eventlog files. " +
+ "This controls each scan process to be completed within a reasonable time, and such " +
+ "prevent the initial scan from running too long and blocking new eventlog files to " +
+ "be scanned in time in large environments.")
+ .version("3.4.0")
+ .intConf
+ .createWithDefault(1000)
Review Comment:
Make sense and changed to Int.MaxValue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876220043
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -494,6 +495,19 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
}
}
.flatMap { entry => EventLogFileReader(fs, entry) }
+ .filter { reader =>
+ try {
+ reader.modificationTime
+ true
+ } catch {
+ case e: Exception =>
Review Comment:
Could you use more specific exception instead of `java.io.Exception`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #36597: [SPARK-39225][CORE] Support `spark.history.fs.update.batchSize`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36597:
URL: https://github.com/apache/spark/pull/36597#issuecomment-1131292324
BTW, I revised the PR and JIRA title more precisely based on your PR content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876457821
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -41,6 +41,15 @@ private[spark] object History {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
+ val UPDATE_BATCHSIZE = ConfigBuilder("spark.history.fs.update.batchSize")
+ .doc("Specifies the batch size for updating new eventlog files. " +
+ "This controls each scan process to be completed within a reasonable time, and such " +
+ "prevent the initial scan from running too long and blocking new eventlog files to " +
+ "be scanned in time in large environments.")
+ .version("3.4.0")
+ .intConf
+ .createWithDefault(1000)
Review Comment:
In addition, the default value should be `Int.MaxValue` to be safe, @hai-tao-1 .
For the new configuration, we disable it by default to be safe at the initial release. We can reduce it at the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876457322
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -41,6 +41,15 @@ private[spark] object History {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
+ val UPDATE_BATCHSIZE = ConfigBuilder("spark.history.fs.update.batchSize")
+ .doc("Specifies the batch size for updating new eventlog files. " +
+ "This controls each scan process to be completed within a reasonable time, and such " +
+ "prevent the initial scan from running too long and blocking new eventlog files to " +
+ "be scanned in time in large environments.")
+ .version("3.4.0")
+ .intConf
Review Comment:
We need a validation. IIUC, this should be positive to be valid. Do you have any other validity check logic?
```scala
.checkValue(s => s > 0, "The batch size should be positive")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876221706
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -494,6 +495,19 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
}
}
.flatMap { entry => EventLogFileReader(fs, entry) }
+ .filter { reader =>
+ try {
+ reader.modificationTime
+ true
+ } catch {
+ case e: Exception =>
+ logError("Exception in getting modificationTime of " + reader.rootPath.getName + ".")
+ false
+ }
+ }
+ .sortWith { case (entry1, entry2) =>
+ entry1.modificationTime > entry2.modificationTime
Review Comment:
This looks orthogonal to `BatchSize` feature. I'd like to recommend you to spin off this `sorting` contribution into another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hai-tao-1 commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
hai-tao-1 commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876510263
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -474,6 +474,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
* applications that haven't been updated since last time the logs were checked.
Review Comment:
Added description explaining the batchSize logic and purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hai-tao-1 commented on pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
hai-tao-1 commented on PR #36597:
URL: https://github.com/apache/spark/pull/36597#issuecomment-1130961143
> Here is the second round review.
>
> * We need a test case for the configuration. Please check the corner cases especially.
> * We need to add a config validation logic.
@dongjoon-hyun I added checkValue which should have a positive value. Is this enough for above two requests? If not, could you add a bit details. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #36597: [SPARK-39225][CORE] Support `spark.history.fs.update.batchSize`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #36597: [SPARK-39225][CORE] Support `spark.history.fs.update.batchSize`
URL: https://github.com/apache/spark/pull/36597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hai-tao-1 commented on pull request #36597: [SPARK-39225][CORE] Support `spark.history.fs.update.batchSize`
Posted by GitBox <gi...@apache.org>.
hai-tao-1 commented on PR #36597:
URL: https://github.com/apache/spark/pull/36597#issuecomment-1131988355
> Thank you for updates, @hai-tao-1 . Yes, the only remaining comment is the test case.
>
> > We need a test case for the configuration. Please check the corner cases especially.
@dongjoon-hyun Added unit test test("SPARK-39225: Support spark.history.fs.update.batchSize").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876220473
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -494,6 +495,19 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
}
}
.flatMap { entry => EventLogFileReader(fs, entry) }
+ .filter { reader =>
+ try {
+ reader.modificationTime
+ true
+ } catch {
+ case e: Exception =>
+ logError("Exception in getting modificationTime of " + reader.rootPath.getName + ".")
Review Comment:
`logError` is too alarming to the users. I guess `logInfo` would be enough because this will be a new message to the users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hai-tao-1 commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
hai-tao-1 commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876412748
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -494,6 +495,19 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
}
}
.flatMap { entry => EventLogFileReader(fs, entry) }
+ .filter { reader =>
+ try {
+ reader.modificationTime
+ true
+ } catch {
+ case e: Exception =>
+ logError("Exception in getting modificationTime of " + reader.rootPath.getName + ".")
Review Comment:
changed to logInfo as sugested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hai-tao-1 commented on a diff in pull request #36597: [SPARK-39225][CORE] History Server Update BatchSize
Posted by GitBox <gi...@apache.org>.
hai-tao-1 commented on code in PR #36597:
URL: https://github.com/apache/spark/pull/36597#discussion_r876509593
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -41,6 +41,15 @@ private[spark] object History {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
+ val UPDATE_BATCHSIZE = ConfigBuilder("spark.history.fs.update.batchSize")
+ .doc("Specifies the batch size for updating new eventlog files. " +
+ "This controls each scan process to be completed within a reasonable time, and such " +
+ "prevent the initial scan from running too long and blocking new eventlog files to " +
+ "be scanned in time in large environments.")
+ .version("3.4.0")
+ .intConf
Review Comment:
Added checkValue for a positive. A positive value should be enough.
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -41,6 +41,15 @@ private[spark] object History {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
+ val UPDATE_BATCHSIZE = ConfigBuilder("spark.history.fs.update.batchSize")
+ .doc("Specifies the batch size for updating new eventlog files. " +
+ "This controls each scan process to be completed within a reasonable time, and such " +
+ "prevent the initial scan from running too long and blocking new eventlog files to " +
+ "be scanned in time in large environments.")
+ .version("3.4.0")
+ .intConf
Review Comment:
Added checkValue for a positive value. A positive value should be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #36597: [SPARK-39225][CORE] Support `spark.history.fs.update.batchSize`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36597:
URL: https://github.com/apache/spark/pull/36597#issuecomment-1132159846
Merged to master. I added you to the Apache Spark contributor group and assigned SPARK-39225 to you, @hai-tao-1 .
Welcome to the Apache Spark community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org