You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/01 07:45:38 UTC
[GitHub] [spark] beliefer opened a new pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
beliefer opened a new pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751
### What changes were proposed in this pull request?
1.Add version information to the configuration of `History`.
2.Update the docs of `History`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.history.fs.logDirectory | 1.1.0 | SPARK-1768 | 21ddd7d1e9f8e2a726427f32422c31706a20ba3f#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.safemodeCheck.interval | 1.6.0 | SPARK-11020 | cf04fdfe71abc395163a625cc1f99ec5e54cc07e#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.update.interval | 1.4.0 | SPARK-6046 | 4527761bcd6501c362baf2780905a0018b9a74ba#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.enabled | 1.3.0 | SPARK-3562 | 8942b522d8a3269a2a357e3a274ed4b3e66ebdde#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e | Branch branch-1.3 does not exist, exists in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml
spark.history.fs.cleaner.interval | 1.4.0 | SPARK-5933 | 1991337336596f94698e79c2366f065c374128ab#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxAge | 1.4.0 | SPARK-5933 | 1991337336596f94698e79c2366f065c374128ab#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxNum | 3.0.0 | SPARK-28294 | bbc2be4f425c4c26450e1bf21db407e81046ce21#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.store.path | 2.3.0 | SPARK-20642 | 74daf622de4e534d5a5929b424a6e836850eefad#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.store.maxDiskUsage | 2.3.0 | SPARK-20654 | 8b497046c647a21bbed1bdfbdcb176745a1d5cd5#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.ui.port | 1.0.0 | SPARK-1276 | 9ae80bf9bd3e4da7443af97b41fe26aa5d35d70b#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.fs.inProgressOptimization.enabled | 2.4.0 | SPARK-6951 | 653fe02415a537299e15f92b56045569864b6183#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.fs.endEventReparseChunkSize | 2.4.0 | SPARK-6951 | 653fe02415a537299e15f92b56045569864b6183#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.fs.eventLog.rolling.maxFilesToRetain | 3.0.0 | SPARK-30481 | a2fe73b83c0e7c61d1c83b236565a71e3d005a71#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.eventLog.rolling.compaction.score.threshold | 3.0.0 | SPARK-30481 | a2fe73b83c0e7c61d1c83b236565a71e3d005a71#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.driverlog.cleaner.enabled | 3.0.0 | SPARK-25118 | 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.driverlog.cleaner.interval | 3.0.0 | SPARK-25118 | 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.driverlog.cleaner.maxAge | 3.0.0 | SPARK-25118 | 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.ui.acls.enable | 1.0.1 | Spark 1489 | c8dd13221215275948b1a6913192d40e0c8cbadd#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.ui.admin.acls | 2.1.1 | SPARK-19033 | 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.ui.admin.acls.groups | 2.1.1 | SPARK-19033 | 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.numReplayThreads | 2.0.0 | SPARK-13988 | 6fdd0e32a6c3fdce1f3f7e1f8d252af05c419f7b#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.retainedApplications | 1.0.0 | SPARK-1276 | 9ae80bf9bd3e4da7443af97b41fe26aa5d35d70b#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.provider | 1.1.0 | SPARK-1768 | 21ddd7d1e9f8e2a726427f32422c31706a20ba3f#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.kerberos.enabled | 1.0.1 | Spark-1490 | 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.kerberos.principal | 1.0.1 | Spark-1490 | 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.kerberos.keytab | 1.0.1 | Spark-1490 | 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.custom.executor.log.url | 3.0.0 | SPARK-26311 | ae5b2a6a92be4986ef5b8062d7fb59318cff6430#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.custom.executor.log.url.applyIncompleteApplication | 3.0.0 | SPARK-26311 | ae5b2a6a92be4986ef5b8062d7fb59318cff6430#diff-6bddeb5e25239974fc13db66266b167b |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085418
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")
val MAX_LOG_NUM = ConfigBuilder("spark.history.fs.cleaner.maxNum")
.doc("The maximum number of log files in the event log directory.")
+ .version("3.0.0")
.intConf
.createWithDefault(Int.MaxValue)
val LOCAL_STORE_DIR = ConfigBuilder("spark.history.store.path")
.doc("Local directory where to cache application history information. By default this is " +
"not set, meaning all history information will be kept in memory.")
+ .version("2.3.0")
.stringConf
.createOptional
val MAX_LOCAL_DISK_USAGE = ConfigBuilder("spark.history.store.maxDiskUsage")
+ .version("2.3.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10g")
val HISTORY_SERVER_UI_PORT = ConfigBuilder("spark.history.ui.port")
.doc("Web UI port to bind Spark History Server")
+ .version("1.0.0")
Review comment:
SPARK-1276, commit ID: 9ae80bf9bd3e4da7443af97b41fe26aa5d35d70b#diff-b49b5b9c31ddb36a9061004b5b723058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085683
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
+ .version("2.0.0")
Review comment:
SPARK-13988, commit ID: 6fdd0e32a6c3fdce1f3f7e1f8d252af05c419f7b#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085607
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
Review comment:
SPARK-25118, commit ID: 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593215749
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593070182
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23883/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC]
Add version information to the configuration of History
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593180265
**[Test build #119145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119145/testReport)** for PR 27751 at commit [`52c7e28`](https://github.com/apache/spark/commit/52c7e28882b64ae61aebe7f1dfac6754c2ac3a9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593215749
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC]
Add version information to the configuration of History
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593067290
**[Test build #119139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119139/testReport)** for PR 27751 at commit [`68a0aa1`](https://github.com/apache/spark/commit/68a0aa1c0313a644269bc409f1a953e47e6179c5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593067375
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23881/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080872
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085231
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
Review comment:
SPARK-11020, commit ID: cf04fdfe71abc395163a625cc1f99ec5e54cc07e#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085850
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
+ .version("2.0.0")
.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)
val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
+ .version("1.0.0")
.intConf
.createWithDefault(50)
val PROVIDER = ConfigBuilder("spark.history.provider")
+ .version("1.1.0")
.stringConf
.createOptional
val KERBEROS_ENABLED = ConfigBuilder("spark.history.kerberos.enabled")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val KERBEROS_PRINCIPAL = ConfigBuilder("spark.history.kerberos.principal")
+ .version("1.0.1")
.stringConf
.createOptional
val KERBEROS_KEYTAB = ConfigBuilder("spark.history.kerberos.keytab")
+ .version("1.0.1")
Review comment:
Spark-1490, commit ID: 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085565
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
Review comment:
SPARK-25118, commit ID: 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386087573
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.4.0")
Review comment:
SPARK-3562, commit ID: 8942b522d8a3269a2a357e3a274ed4b3e66ebdde#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
This item does not in branch-1.3, in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593180437
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593180437
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085623
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
Review comment:
Spark 1489, commit ID: c8dd13221215275948b1a6913192d40e0c8cbadd#diff-b49b5b9c31ddb36a9061004b5b723058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085638
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
Review comment:
SPARK-19033, commit ID: 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085399
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")
val MAX_LOG_NUM = ConfigBuilder("spark.history.fs.cleaner.maxNum")
.doc("The maximum number of log files in the event log directory.")
+ .version("3.0.0")
.intConf
.createWithDefault(Int.MaxValue)
val LOCAL_STORE_DIR = ConfigBuilder("spark.history.store.path")
.doc("Local directory where to cache application history information. By default this is " +
"not set, meaning all history information will be kept in memory.")
+ .version("2.3.0")
.stringConf
.createOptional
val MAX_LOCAL_DISK_USAGE = ConfigBuilder("spark.history.store.maxDiskUsage")
+ .version("2.3.0")
Review comment:
SPARK-20654, commit ID: 8b497046c647a21bbed1bdfbdcb176745a1d5cd5#diff-19f35f981fdc5b0a46f070b879a9a9fc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085298
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
Review comment:
SPARK-3562, commit ID: 8942b522d8a3269a2a357e3a274ed4b3e66ebdde#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
This item does not in branch-1.3, in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593070653
**[Test build #119141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119141/testReport)** for PR 27751 at commit [`52c7e28`](https://github.com/apache/spark/commit/52c7e28882b64ae61aebe7f1dfac6754c2ac3a9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593180441
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23887/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593215755
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119145/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080873
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119139/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593070182
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23883/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593067290
**[Test build #119139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119139/testReport)** for PR 27751 at commit [`68a0aa1`](https://github.com/apache/spark/commit/68a0aa1c0313a644269bc409f1a953e47e6179c5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085370
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")
val MAX_LOG_NUM = ConfigBuilder("spark.history.fs.cleaner.maxNum")
.doc("The maximum number of log files in the event log directory.")
+ .version("3.0.0")
Review comment:
SPARK-28294, commit ID: bbc2be4f425c4c26450e1bf21db407e81046ce21#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085517
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -90,6 +102,7 @@ private[spark] object History {
"By default, all event log files will be retained. Please set the configuration " +
s"and ${EVENT_LOG_ROLLING_MAX_FILE_SIZE.key} accordingly if you want to control " +
"the overall size of event log files.")
+ .version("3.0.0")
Review comment:
SPARK-30481, commit ID: a2fe73b83c0e7c61d1c83b236565a71e3d005a71#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593070181
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085342
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
Review comment:
SPARK-5933, commit ID: 1991337336596f94698e79c2366f065c374128ab#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085789
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
+ .version("2.0.0")
.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)
val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
+ .version("1.0.0")
.intConf
.createWithDefault(50)
val PROVIDER = ConfigBuilder("spark.history.provider")
+ .version("1.1.0")
Review comment:
SPARK-1768, commit ID: 21ddd7d1e9f8e2a726427f32422c31706a20ba3f#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593070181
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085298
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
Review comment:
SPARK-3562, commit ID: 8942b522d8a3269a2a357e3a274ed4b3e66ebdde#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593215755
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119145/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593180441
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23887/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085591
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
Review comment:
SPARK-25118, commit ID: 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080873
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119139/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085665
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
Review comment:
SPARK-19033, commit ID: 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593179564
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085502
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -81,6 +92,7 @@ private[spark] object History {
.doc("How many bytes to parse at the end of log files looking for the end event. " +
"This is used to speed up generation of application listings by skipping unnecessary " +
"parts of event log files. It can be disabled by setting this config to 0.")
+ .version("2.4.0")
Review comment:
SPARK-6951, commit ID: 653fe02415a537299e15f92b56045569864b6183#diff-19f35f981fdc5b0a46f070b879a9a9fc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593067375
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23881/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085885
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -165,6 +192,7 @@ private[spark] object History {
s"${CUSTOM_EXECUTOR_LOG_URL.key}, to incomplete application as well. " +
"Even if this is true, this still only affects the behavior of the history server, " +
"not running spark applications.")
+ .version("3.0.0")
Review comment:
SPARK-26311, commit ID: ae5b2a6a92be4986ef5b8062d7fb59318cff6430#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-609306424
Merged to branch-3.0 too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080872
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593067374
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC]
Add version information to the configuration of History
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593215149
**[Test build #119145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119145/testReport)** for PR 27751 at commit [`52c7e28`](https://github.com/apache/spark/commit/52c7e28882b64ae61aebe7f1dfac6754c2ac3a9c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593335285
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC]
Add version information to the configuration of History
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080695
**[Test build #119139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119139/testReport)** for PR 27751 at commit [`68a0aa1`](https://github.com/apache/spark/commit/68a0aa1c0313a644269bc409f1a953e47e6179c5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085804
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
+ .version("2.0.0")
.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)
val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
+ .version("1.0.0")
.intConf
.createWithDefault(50)
val PROVIDER = ConfigBuilder("spark.history.provider")
+ .version("1.1.0")
.stringConf
.createOptional
val KERBEROS_ENABLED = ConfigBuilder("spark.history.kerberos.enabled")
+ .version("1.0.1")
Review comment:
Spark-1490, commit ID: 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593067374
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085729
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
+ .version("2.0.0")
.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)
val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
+ .version("1.0.0")
Review comment:
SPARK-1276, commit ID: 9ae80bf9bd3e4da7443af97b41fe26aa5d35d70b#diff-b49b5b9c31ddb36a9061004b5b723058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC]
Add version information to the configuration of History
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593070653
**[Test build #119141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119141/testReport)** for PR 27751 at commit [`52c7e28`](https://github.com/apache/spark/commit/52c7e28882b64ae61aebe7f1dfac6754c2ac3a9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085867
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -156,6 +182,7 @@ private[spark] object History {
"some path variables via patterns which can vary on cluster manager. Please check the " +
"documentation for your cluster manager to see which patterns are supported, if any. " +
"This configuration has no effect on a live application, it only affects the history server.")
+ .version("3.0.0")
Review comment:
SPARK-26311, commit ID: ae5b2a6a92be4986ef5b8062d7fb59318cff6430#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085215
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
Review comment:
SPARK-1768，commit ID: 21ddd7d1e9f8e2a726427f32422c31706a20ba3f#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085263
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
Review comment:
SPARK-6046, commit ID: 4527761bcd6501c362baf2780905a0018b9a74ba#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080859
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080861
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119141/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085378
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")
val MAX_LOG_NUM = ConfigBuilder("spark.history.fs.cleaner.maxNum")
.doc("The maximum number of log files in the event log directory.")
+ .version("3.0.0")
.intConf
.createWithDefault(Int.MaxValue)
val LOCAL_STORE_DIR = ConfigBuilder("spark.history.store.path")
.doc("Local directory where to cache application history information. By default this is " +
"not set, meaning all history information will be kept in memory.")
+ .version("2.3.0")
Review comment:
SPARK-20642, commit ID: 74daf622de4e534d5a5929b424a6e836850eefad#diff-19f35f981fdc5b0a46f070b879a9a9fc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080861
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119141/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085823
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
+ .version("2.0.0")
.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)
val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
+ .version("1.0.0")
.intConf
.createWithDefault(50)
val PROVIDER = ConfigBuilder("spark.history.provider")
+ .version("1.1.0")
.stringConf
.createOptional
val KERBEROS_ENABLED = ConfigBuilder("spark.history.kerberos.enabled")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val KERBEROS_PRINCIPAL = ConfigBuilder("spark.history.kerberos.principal")
+ .version("1.0.1")
Review comment:
Spark-1490, commit ID: 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593180265
**[Test build #119145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119145/testReport)** for PR 27751 at commit [`52c7e28`](https://github.com/apache/spark/commit/52c7e28882b64ae61aebe7f1dfac6754c2ac3a9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085528
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
Review comment:
SPARK-30481, commit ID: a2fe73b83c0e7c61d1c83b236565a71e3d005a71#diff-6bddeb5e25239974fc13db66266b167b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC]
Add version information to the configuration of History
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080775
**[Test build #119141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119141/testReport)** for PR 27751 at commit [`52c7e28`](https://github.com/apache/spark/commit/52c7e28882b64ae61aebe7f1dfac6754c2ac3a9c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085455
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")
val MAX_LOG_NUM = ConfigBuilder("spark.history.fs.cleaner.maxNum")
.doc("The maximum number of log files in the event log directory.")
+ .version("3.0.0")
.intConf
.createWithDefault(Int.MaxValue)
val LOCAL_STORE_DIR = ConfigBuilder("spark.history.store.path")
.doc("Local directory where to cache application history information. By default this is " +
"not set, meaning all history information will be kept in memory.")
+ .version("2.3.0")
.stringConf
.createOptional
val MAX_LOCAL_DISK_USAGE = ConfigBuilder("spark.history.store.maxDiskUsage")
+ .version("2.3.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10g")
val HISTORY_SERVER_UI_PORT = ConfigBuilder("spark.history.ui.port")
.doc("Web UI port to bind Spark History Server")
+ .version("1.0.0")
.intConf
.createWithDefault(18080)
val FAST_IN_PROGRESS_PARSING =
ConfigBuilder("spark.history.fs.inProgressOptimization.enabled")
.doc("Enable optimized handling of in-progress logs. This option may leave finished " +
"applications that fail to rename their event logs listed as in-progress.")
+ .version("2.4.0")
Review comment:
SPARK-6951, commit ID: 653fe02415a537299e15f92b56045569864b6183#diff-19f35f981fdc5b0a46f070b879a9a9fc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#issuecomment-593080859
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27751:
[SPARK-30891][CORE][DOC] Add version information to the configuration of
History
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
URL: https://github.com/apache/spark/pull/27751#discussion_r386085351
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.3.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
+ .version("1.4.0")
Review comment:
SPARK-5933, commit ID: 1991337336596f94698e79c2366f065c374128ab#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org