You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/29 23:39:39 UTC
[GitHub] [spark] baohe-zhang opened a new pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang opened a new pull request #28412:
URL: https://github.com/apache/spark/pull/28412
### What changes were proposed in this pull request?
Add a new class HybridKVStore to make the history server faster when loading event files. When rebuilding the application state from event logs, HybridKVStore will first write data to an in-memory store and having a background thread that keeps pushing the change to levelDB.
### Why are the changes needed?
HybridKVStore can significantly reduce the event logs loading time, especially for large log files. The table below shows some test results I ran on mac os.
kvstore type / log size | 100m | 200m | 500m | 1g | 2g
-- | -- | -- | -- | -- | --
HybridKVStore | 5s to parse, 7s(include the parsing time) to switch to leveldb | 6s to parse, 10s to switch to leveldb | 15s to parse, 23s to switch to leveldb | 23s to parse, 40s to switch to leveldb | 37s to parse, 73s to switch to leveldb
LevelDB | 12s to parse | 19s to parse | 43s to parse | 69s to parse | 124s to parse
### Does this PR introduce any user-facing change?
This PR adds a new config "spark.history.store.hybridKVStore.enabled".
### How was this patch tested?
A test suite for HybridKVStore is added. I also manually tested it on 3.0.1 on mac os.
This is a follow-up for the work done by Hieu Huynh in 2019.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656908439
**[Test build #125635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125635/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644485557
**[Test build #124078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124078/testReport)** for PR 28412 at commit [`3412d29`](https://github.com/apache/spark/commit/3412d294a76a7348377f4d1a87327dc4fe510e96).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644174398
**[Test build #124053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124053/testReport)** for PR 28412 at commit [`76dbd18`](https://github.com/apache/spark/commit/76dbd184b659c2eeccba56f8201d470d939fceb4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r441165290
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
comparison between memory usage and disk usage for some randomly chosen 2.4 logs, parsed by 2.4 SHS
log size | 1.0g | 1.26g | 1.16g | 1.11g
-- | -- | -- | -- | --
memory usage | 97.8m | 232m | 53.1m | 31.4m
leveldb filesize | 183M | 368m | 220m | 99m
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440955307
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
I will add comments about this function and test compressed files later. So far I think this estimation is conservative and safe. For large uncompressed log files, the memory and disk usage are usually much smaller than the estimated usage(half of the log file size). I measured some log files with size ~1g that I randomly picked from production-use spark jobs.
log filesize | 1.0g | 1.26g | 1.16g | 1.11g
-- | -- | -- | -- | --
memory usage | 113m | 149m | 63m | 46m
leveldb filesize | 175m | 211m | 42m | 77m
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624423684
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122332/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641732044
**[Test build #123719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123719/testReport)** for PR 28412 at commit [`1e514b9`](https://github.com/apache/spark/commit/1e514b910a56b719a08d6f7a7689a2a53dcc06a5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-657921314
@tgravescs Do you plan another round of review, or OK as it is?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654561949
**[Test build #125163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125163/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420531106
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
Review comment:
1 is addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621660613
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-634674147
@HeartSaVioR @tgravescs @vanzin do you mind taking a look at this pr again thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638627240
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654533150
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644486208
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-642152258
**[Test build #123770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123770/testReport)** for PR 28412 at commit [`76dbd18`](https://github.com/apache/spark/commit/76dbd184b659c2eeccba56f8201d470d939fceb4).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-655125113
@baohe-zhang
Thanks for the update. This is really helpful. So the small event log file shows there's a chance the ratio can be beyond 1/2.
Personally it seems OK to simply allow such case, as these small event logs can be migrated to the LevelDB KV store fast enough, and the memory usage would hold shortly. That said, it'd be simpler if we can just go through with LevelDB KV store for small event logs, no need to run background thread per app log.
Either is fine for me. For latter the lower bound should be configured against uncompressed log, and we should do the similar estimation on compressed log before applying. That's another tricky one, but probably good to have. (I wouldn't run background thread for event log which can be loaded in a couple of seconds)
@tgravescs WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638627349
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638627353
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123520/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641777310
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654837665
**[Test build #125219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125219/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-636833992
**[Test build #123375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123375/testReport)** for PR 28412 at commit [`a796231`](https://github.com/apache/spark/commit/a796231c82ca5b2206b25d4bf6bea951ea3ee631).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641024447
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656418437
Adding batch update is a perf improvement - not required for the
functionality being added.
I mentioned it given I did it recently and similar snippet was pretty much
exactly what I had replaced :-)
On Thu, Jul 9, 2020 at 9:04 AM Baohe Zhang <no...@github.com> wrote:
> *@baohe-zhang* commented on this pull request.
> ------------------------------
>
> In core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
> <https://github.com/apache/spark/pull/28412#discussion_r452327580>:
>
> > + */
> + def switchToLevelDB(
> + listener: HybridStore.SwitchToLevelDBListener,
> + appId: String,
> + attemptId: Option[String]): Unit = {
> + if (closed.get) {
> + return
> + }
> +
> + backgroundThread = new Thread(() => {
> + try {
> + for (klass <- klassMap.keys().asScala) {
> + val it = inMemoryStore.view(klass).closeableIterator()
> + while (it.hasNext()) {
> + levelDB.write(it.next())
> + }
>
> Make sense. I am testing it on the mac which has SSD and the disk is not
> busy. I think the improvement might be more obvious on HDD or busy disk.
> @HeartSaVioR <https://github.com/HeartSaVioR> @tgravescs
> <https://github.com/tgravescs> Do we need to add batch write support for
> leveldb on this pr?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/28412#discussion_r452327580>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAMETFA74MHMBGAZ5XWWC43R2XTCFANCNFSM4MUQYGFQ>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658211580
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437811279
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
Review comment:
yeah, that makes sense, I will update the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656689389
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125598/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-642054235
**[Test build #123770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123770/testReport)** for PR 28412 at commit [`76dbd18`](https://github.com/apache/spark/commit/76dbd184b659c2eeccba56f8201d470d939fceb4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624425131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656444886
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641806917
**[Test build #123742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123742/testReport)** for PR 28412 at commit [`0d36074`](https://github.com/apache/spark/commit/0d360741d6c8cf2a1f4b597a860636e512226875).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644430538
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621592632
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437577822
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
+ case Some(e) =>
+ listener.onSwitchToLevelDBFail(e)
Review comment:
done.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641701635
**[Test build #123720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123720/testReport)** for PR 28412 at commit [`d6c7d98`](https://github.com/apache/spark/commit/d6c7d988bd9e39caebb9a33f8c01ee230b6c2a39).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621630002
**[Test build #122105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122105/testReport)** for PR 28412 at commit [`0164960`](https://github.com/apache/spark/commit/01649606321f3ce615fbf8fd06d6854a936593c1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class HybridKVStore implements KVStore `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624423679
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r436235957
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
Review comment:
Given we still have to deal with exception, how about generalizing a bit more, like allowing to write on in-memory DB but capturing all of the write operations between the time the replay is done and the time we just shift to level DB?
You may feel it sounds as back and forth - sorry I didn't realize there are additional writes after replaying. But it only applies to the case during migrating so would be still simpler than before, and we can get rid of assertion on being "read-only" with exception.
I'm also OK with current state - eventually I'll try to deal with that afterwards.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437789267
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
+ if (hybridStoreEnabled) {
+ try {
+ return createHybridStore(dm, appId, attempt, metadata)
+ } catch {
+ case e: Exception =>
+ logInfo(s"Failed to create hybrid store for $appId/${attempt.info.attemptId}." +
Review comment:
Either is fine for me personally. Let's hear more voices.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644070017
**[Test build #124053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124053/testReport)** for PR 28412 at commit [`76dbd18`](https://github.com/apache/spark/commit/76dbd184b659c2eeccba56f8201d470d939fceb4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437529862
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,71 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ val memoryUsage = approximateMemoryUsage(reader.totalSize, isCompressed)
+ if (currentInMemoryStoreUsage.get + memoryUsage > maxInMemoryStoreUsage) {
+ throw new IllegalStateException("Not enough in-memory storage to create hybrid store.")
+ }
+ currentInMemoryStoreUsage.addAndGet(memoryUsage)
+ logInfo(s"Attempt creating hybrid store to parse $appId / ${attempt.info.attemptId}. " +
+ s"Requested ${Utils.bytesToString(memoryUsage)} in-memory storage quota.")
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ store.setCachedQuantileKlass(classOf[CachedQuantile])
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchingToLevelDB(new HybridStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ store.setLevelDB(KVUtils.open(newStorePath, metadata))
+ currentInMemoryStoreUsage.addAndGet(-memoryUsage)
+ logInfo(s"Completely switched to leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}. " +
+ s"Released ${Utils.bytesToString(memoryUsage)} in-memory storage quota.")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}", e)
+ levelDB.close()
+ if (!isLeaseRolledBack.getAndSet(true)) {
+ lease.rollback()
+ }
+ }
+ })
+
+ store
+ } catch {
+ case e: Exception =>
+ store.close()
+ currentInMemoryStoreUsage.addAndGet(-memoryUsage)
Review comment:
Ah missed it thanks
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
+ if (hybridStoreEnabled) {
+ try {
+ return createHybridStore(dm, appId, attempt, metadata)
+ } catch {
+ case e: Exception =>
+ logInfo(s"Failed to create hybrid store for $appId/${attempt.info.attemptId}." +
Review comment:
Should this be warning? may be having it as error would not be neccessary as leveldb would work?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
Review comment:
+1
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
+ override def onSwitchToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ store.setLevelDB(KVUtils.open(newStorePath, metadata))
+ memoryManager.release(appId, attempt.info.attemptId)
+ logInfo(s"Completely switched to LevelDB for app $appId / ${attempt.info.attemptId}.")
+ }
+
+ override def onSwitchToLevelDBFail(e: Exception): Unit = {
+ levelDB.close()
+ if (!isLeaseRolledBack.getAndSet(true)) {
+ lease.rollback()
+ }
Review comment:
Yeah LeaseRolledBack seems uneccessary
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
+ override def onSwitchToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ store.setLevelDB(KVUtils.open(newStorePath, metadata))
+ memoryManager.release(appId, attempt.info.attemptId)
+ logInfo(s"Completely switched to LevelDB for app $appId / ${attempt.info.attemptId}.")
+ }
+
+ override def onSwitchToLevelDBFail(e: Exception): Unit = {
+ levelDB.close()
+ if (!isLeaseRolledBack.getAndSet(true)) {
+ lease.rollback()
+ }
Review comment:
Yeah isLeaseRolledBack seems uneccessary
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
Review comment:
Do we expect a write after a background thread has started? We might want to throw an IIlegalStateException
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621630288
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621660096
**[Test build #122119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122119/testReport)** for PR 28412 at commit [`7e3c39e`](https://github.com/apache/spark/commit/7e3c39eda716d51c13e594487443262c515b07d0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-642153243
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658211083
**[Test build #125841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125841/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437845789
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
Review comment:
Friendly reminder about the comment here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621526046
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420472617
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
+ }
+ })
+
+ rebuildAppStore(s, reader, attempt.info.lastUpdated.getTime())
+ s.stopBackgroundThreadAndSwitchToLevelDB()
+ store = s
+ } catch {
+ case _: IOException if !retried =>
+ // compaction may touch the file(s) which app rebuild wants to read
+ // compaction wouldn't run in short interval, so try again...
+ logWarning(s"Exception occurred while rebuilding app $appId - trying again...")
+ lease.rollback()
Review comment:
I missed the instance will be dereferenced, my bad.
It might be still ideal to clean up the instance explicitly, as the instance may keep the huge memory usages. Please note that this can happen in the middle of processing.
(The previous logic may not be complete to clean up the instance as well, so just 2 cents.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624476402
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656444886
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621535327
So you mean for loading 1Gb of eventlogs, the HybridKVStore takes 63s while the current LevelDB takes 69s ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641701924
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644482579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437837726
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1263,17 +1241,34 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
"trying again...")
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
retried = true
-
case e: Exception =>
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
throw e
}
}
+ // Create a LevelDB and start a background thread to dump data to LevelDB
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
Review comment:
ok, will update it. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437846179
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ closed.set(true)
+
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+
+ try {
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ listener.onSwitchToLevelDBSuccess()
+ shouldUseInMemoryStore.set(false)
+ inMemoryStore.close()
+ } catch {
+ case e: Exception =>
+ listener.onSwitchToLevelDBFail(e)
+ }
+ })
+ backgroundThread.setDaemon(true)
+ backgroundThread.setName("hybridstore-switch-to-leveldb")
Review comment:
Maybe better to track if we add the unique ID like application ID + attempt ID in thread name. To reduce length I guess `hybridstore-` from the current string is enough as there's only one kind of thread here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656909607
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624070783
concept seems good, a couple of high level questions without me having looked at the detailed code.
I assume the number of threads to read is still spark.history.fs.numReplayThreads?
It looks like you only have one thread to write to levelDB. Is this enough for it to not be backed up? Lets say we have a very active history server and user loading a lot of large files, does the memory balloon before it can be flushed to levelDB?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641765619
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621543329
@baohe-zhang Good to know the result, it sounds great! cc @rednaxelafx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641809181
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644070471
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-640924422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-640924005
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644175741
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638538734
Oh OK there's an exception... My bad. Thanks for noticing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656593776
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641777310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644486208
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624423178
**[Test build #122332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122332/testReport)** for PR 28412 at commit [`e6707bd`](https://github.com/apache/spark/commit/e6707bd61e747131b118b8326d01cba159df614a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r436151953
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
Review comment:
Why do we need to handle cachedQuantileQueue specially?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437581083
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
Review comment:
private added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654838940
thanks, for testing those combinations, that shows the number of jobs doesn't affect the memory usage much, but if I'm not mistaken in all the cases you have 400,000 tasks. My main concern was the memory usage with small number of tasks total - say 400 or 1000? Does it still stay to the same ratio or do we way over estimate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638627332
**[Test build #123520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123520/testReport)** for PR 28412 at commit [`c6b5392`](https://github.com/apache/spark/commit/c6b539218465f94f82cd7562a99b8792640096e2).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638647669
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123521/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656822532
**[Test build #125635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125635/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r451838037
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
Thanks, this is helpful!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440918495
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
the difference is though that is disk usage, not memory usage. If you go over disk usage temporarily generally not as big of a deal if you go over memory usage. I'm wondering for instance if the codec is Zstd could the memory usage be much more if it compressed it very well. I'm not sure a good solution to this at the moment other then perhaps make the multiplication factor configurable, but I'm fine with waiting and seeing how it runs and if you have tested some cases and seems close that is fine to start with.
Please add the comments saying you copied from the HistoryServerDiskManager and put similar comments that function has about it being approximate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r451795960
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
If this helps, this is what I had written up:
```
@Override
public void write(List<?> values) throws Exception {
Preconditions.checkArgument(values != null && !values.isEmpty(),
"Non-empty values required.");
// Group by class, in case there are values from different classes in the iterator
// Typical usecase is for this to be a single class.
for (Map.Entry<? extends Class<?>, ? extends List<?>> entry :
values.stream().collect(Collectors.groupingBy(Object::getClass)).entrySet()) {
final Iterator<?> valueIter = entry.getValue().iterator();
final Iterator<byte[]> serializedValueIter;
{
// deserialize outside synchronized block
List<byte[]> list = new ArrayList<>(entry.getValue().size());
for (Object value : entry.getValue()) {
list.add(serializer.serialize(value));
}
serializedValueIter = list.iterator();
}
final Class<?> valueClass = entry.getKey();
final LevelDBTypeInfo ti = getTypeInfo(valueClass);
// Batching updates per type
synchronized (ti) {
final LevelDBTypeInfo.Index naturalIndex = ti.naturalIndex();
final Collection<LevelDBTypeInfo.Index> indices = ti.indices();
try (WriteBatch batch = db().createWriteBatch()) {
while (valueIter.hasNext()) {
final Object value = valueIter.next();
assert serializedValueIter.hasNext();
final byte[] serializedObject = serializedValueIter.next();
Object existing;
try {
existing = get(naturalIndex.entityKey(null, value), valueClass);
} catch (NoSuchElementException e) {
existing = null;
}
PrefixCache cache = new PrefixCache(value);
byte[] naturalKey = naturalIndex.toKey(naturalIndex.getValue(value));
for (LevelDBTypeInfo.Index idx : indices) {
byte[] prefix = cache.getPrefix(idx);
idx.add(batch, value, existing, serializedObject, naturalKey, prefix);
}
}
assert !serializedValueIter.hasNext();
db().write(batch);
}
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641776225
**[Test build #123722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123722/testReport)** for PR 28412 at commit [`0d36074`](https://github.com/apache/spark/commit/0d360741d6c8cf2a1f4b597a860636e512226875).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420463955
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
Review comment:
Will address 1. For 2, if switching to leveldb failed, hybrid kvstore will always use in-memory kvstore as it's underlying kvstore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437849260
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
Review comment:
Sorry I missed this comment. Yeah, you are right, join() can throw InterruptedException. I will update the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440499121
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ closed.set(true)
+
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+
+ try {
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ listener.onSwitchToLevelDBSuccess()
+ shouldUseInMemoryStore.set(false)
+ inMemoryStore.close()
+ } catch {
+ case e: Exception =>
+ listener.onSwitchToLevelDBFail(e)
+ }
+ })
+ backgroundThread.setDaemon(true)
+ backgroundThread.setName("hybridstore-switch-to-leveldb")
Review comment:
Addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638626777
**[Test build #123520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123520/testReport)** for PR 28412 at commit [`c6b5392`](https://github.com/apache/spark/commit/c6b539218465f94f82cd7562a99b8792640096e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-636833992
**[Test build #123375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123375/testReport)** for PR 28412 at commit [`a796231`](https://github.com/apache/spark/commit/a796231c82ca5b2206b25d4bf6bea951ea3ee631).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-674948675
@gatorsmile Thanks for reminding me that! I will add unit tests for these HybridStore related PRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-655125113
@baohe-zhang
Thanks for the update. This is really helpful. So the small event log file shows there's a chance the ratio can be beyond 1/2. (There's information only for uncompressed one, but let's deal with compressed logs when we really need.)
Personally it seems OK to simply allow such case, as these small event logs can be migrated to the LevelDB KV store fast enough, and the memory usage would hold shortly. That said, it'd be simpler if we can just go through with LevelDB KV store for small event logs, no need to run background thread per app log.
Either is fine for me. For latter the lower bound should be configured against uncompressed log, and we should do the similar estimation on compressed log before applying. That's another tricky one, but probably good to have. (I wouldn't run background thread for event log which can be loaded in a couple of seconds)
@tgravescs WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656444891
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125522/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658314447
**[Test build #125841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125841/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641809181
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437580889
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
Review comment:
updated to IllegalStateException
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624964828
The idea is similar with HistoryServerDiskManager so makes sense in general. We may need to get concrete answers for these questions to go forward:
1. How we will guarantee these area of memory is used only for Hybrid KV store to prevent OOM? (Or no guard and end users have to deal with providing enough memory on heap?)
2. How to calculate approximate memory usage? I guess it would be safe to assume the approximate size as event log file size, but it would take over huge memory for single app. (That may not be a problem on safety perspective, but pretty less efficient.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624479484
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624415280
Also worth to mention that memory usage should be under control; there's no restriction for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638632997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624439430
@HeartSaVioR One way in my mind is that we can monitor the memory usage of SHS. If the memory usage or event log size exceeds a threshold (e.g, over 50% of Xmx), we can use leveldb to parse event log, instead of hybrid kvstore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-636843886
I'm not missing this PR, but Spark 3.0.0 must be the highest priority as well as this change is non-trivial (800+ lines dealing with multi-threads) I'm not sure I can take a deep look at this soon.
Btw, I haven't changed my voice - personally I still feel it a bit complicated, as I commented earlier on https://github.com/apache/spark/pull/28412#pullrequestreview-405371683. I don't think hybrid KV Store will be used on general purpose, as its purpose is to help to replay the events faster. Could we leverage the fact to simplify the case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654533156
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125136/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437846179
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ closed.set(true)
+
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+
+ try {
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ listener.onSwitchToLevelDBSuccess()
+ shouldUseInMemoryStore.set(false)
+ inMemoryStore.close()
+ } catch {
+ case e: Exception =>
+ listener.onSwitchToLevelDBFail(e)
+ }
+ })
+ backgroundThread.setDaemon(true)
+ backgroundThread.setName("hybridstore-switch-to-leveldb")
Review comment:
Maybe better to track if we add the unique ID like application ID + attempt ID in thread name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r444805076
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
It'd be nice if we experiment with all available compressions for event log and have numbers here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638627240
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644440678
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r436796185
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,71 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ val memoryUsage = approximateMemoryUsage(reader.totalSize, isCompressed)
+ if (currentInMemoryStoreUsage.get + memoryUsage > maxInMemoryStoreUsage) {
+ throw new IllegalStateException("Not enough in-memory storage to create hybrid store.")
+ }
+ currentInMemoryStoreUsage.addAndGet(memoryUsage)
+ logInfo(s"Attempt creating hybrid store to parse $appId / ${attempt.info.attemptId}. " +
+ s"Requested ${Utils.bytesToString(memoryUsage)} in-memory storage quota.")
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ store.setCachedQuantileKlass(classOf[CachedQuantile])
Review comment:
ok, will do that. I am moving it to org.apache.spark.deploy.history module.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,71 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ val memoryUsage = approximateMemoryUsage(reader.totalSize, isCompressed)
+ if (currentInMemoryStoreUsage.get + memoryUsage > maxInMemoryStoreUsage) {
+ throw new IllegalStateException("Not enough in-memory storage to create hybrid store.")
+ }
+ currentInMemoryStoreUsage.addAndGet(memoryUsage)
+ logInfo(s"Attempt creating hybrid store to parse $appId / ${attempt.info.attemptId}. " +
+ s"Requested ${Utils.bytesToString(memoryUsage)} in-memory storage quota.")
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ store.setCachedQuantileKlass(classOf[CachedQuantile])
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchingToLevelDB(new HybridStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ store.setLevelDB(KVUtils.open(newStorePath, metadata))
+ currentInMemoryStoreUsage.addAndGet(-memoryUsage)
+ logInfo(s"Completely switched to leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}. " +
+ s"Released ${Utils.bytesToString(memoryUsage)} in-memory storage quota.")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}", e)
+ levelDB.close()
+ if (!isLeaseRolledBack.getAndSet(true)) {
+ lease.rollback()
+ }
+ }
+ })
+
+ store
+ } catch {
+ case e: Exception =>
+ store.close()
+ currentInMemoryStoreUsage.addAndGet(-memoryUsage)
Review comment:
decrementAndGet() only decrements one, there is no method for decrementAndGet(long n)
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
Review comment:
addressed. removed caution.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
Review comment:
addressed.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type, index, indexedValue);
+ return count;
+ }
+
+ @Override
+ public void close() throws IOException {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join();
+ }
+ if (levelDB != null) {
+ levelDB.close();
+ }
+ if (inMemoryStore != null) {
+ inMemoryStore.close();
+ }
+ } catch (Exception e) {
+ if (e instanceof IOException) {
+ throw (IOException) e;
+ }
+ }
+ }
+
+ @Override
+ public <T> boolean removeAllByIndexValues(
+ Class<T> klass,
+ String index,
+ Collection<?> indexValues) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type, index, indexedValue);
+ return count;
+ }
+
+ @Override
+ public void close() throws IOException {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join();
+ }
+ if (levelDB != null) {
+ levelDB.close();
+ }
+ if (inMemoryStore != null) {
+ inMemoryStore.close();
+ }
+ } catch (Exception e) {
+ if (e instanceof IOException) {
+ throw (IOException) e;
+ }
+ }
+ }
+
+ @Override
+ public <T> boolean removeAllByIndexValues(
+ Class<T> klass,
+ String index,
+ Collection<?> indexValues) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ boolean removed = store.removeAllByIndexValues(klass, index, indexValues);
+ return removed;
+ }
+
+ public void setLevelDB(LevelDB levelDB) {
+ this.levelDB = levelDB;
+ }
+
+ public void setCachedQuantileKlass(Class<?> klass) {
+ this.cachedQuantileKlass = klass;
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ public void switchingToLevelDB(SwitchingToLevelDBListener listener) throws Exception {
+ // A background thread that dumps data to levelDB
+ backgroundThread = new Thread(new Runnable() {
+ public void run() {
+ Exception exceptionCaught = null;
+
+ try {
+ for (Class<?> klass : klassMap.keySet()) {
+ KVStoreIterator<?> it = inMemoryStore.view(klass).closeableIterator();
+ while (it.hasNext()) {
+ levelDB.write(it.next());
+ }
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
Review comment:
removed
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type, index, indexedValue);
+ return count;
+ }
+
+ @Override
+ public void close() throws IOException {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join();
+ }
+ if (levelDB != null) {
+ levelDB.close();
+ }
+ if (inMemoryStore != null) {
+ inMemoryStore.close();
+ }
+ } catch (Exception e) {
+ if (e instanceof IOException) {
+ throw (IOException) e;
+ }
+ }
+ }
+
+ @Override
+ public <T> boolean removeAllByIndexValues(
+ Class<T> klass,
+ String index,
+ Collection<?> indexValues) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ boolean removed = store.removeAllByIndexValues(klass, index, indexValues);
+ return removed;
+ }
+
+ public void setLevelDB(LevelDB levelDB) {
+ this.levelDB = levelDB;
+ }
+
+ public void setCachedQuantileKlass(Class<?> klass) {
+ this.cachedQuantileKlass = klass;
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ public void switchingToLevelDB(SwitchingToLevelDBListener listener) throws Exception {
+ // A background thread that dumps data to levelDB
+ backgroundThread = new Thread(new Runnable() {
+ public void run() {
+ Exception exceptionCaught = null;
+
+ try {
+ for (Class<?> klass : klassMap.keySet()) {
+ KVStoreIterator<?> it = inMemoryStore.view(klass).closeableIterator();
+ while (it.hasNext()) {
+ levelDB.write(it.next());
+ }
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ exceptionCaught = e;
+ }
+
+ if (exceptionCaught == null) {
+ // Switch to levelDB and close inMemoryStore
+ shouldUseInMemoryStore.set(false);
+
+ // Dump CachedQuantile objects to levelDB
+ try {
+ while(cachedQuantileQueue.size() > 0) {
+ levelDB.write(cachedQuantileQueue.poll());
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
Review comment:
I changed the logic of writing CachedQuantile to leveldb.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val MAX_IN_MEMORY_STORE_USAGE = ConfigBuilder("spark.history.store.hybridStore.maxMemoryUsage")
+ .version("3.1.0")
Review comment:
doc added.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
done.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -128,6 +129,10 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
private val storePath = conf.get(LOCAL_STORE_DIR).map(new File(_))
private val fastInProgressParsing = conf.get(FAST_IN_PROGRESS_PARSING)
+ private val hybridStoreEnabled = conf.get(History.HYBRID_STORE_ENABLED)
+ private val maxInMemoryStoreUsage = conf.get(History.MAX_IN_MEMORY_STORE_USAGE)
+ private val currentInMemoryStoreUsage = new java.util.concurrent.atomic.AtomicLong(0L)
Review comment:
a new class HistoryServerMemoryManager is created.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
I think the main purpose is to handle CachedQuantile specifically. CachedQuantile is not on this list, so it won't be dumped to levelDB in the background thread. Right now my strategy is to write CachedQuantile to inMemoryStore and levelDB simultaneously during the transition process (background thread is alive), this strategy seems ok in my manual test.
CachedQuantile is quite special, it will only be added to kvstore after rebuildAppStore() finishes. that means if we don't handle it specially it will be added to inMemoryStore when the background thread is dumping data(including CachedQuantile objects) to leveldb and hence cause conflicts.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
ok, I will use a reject-list and add CachedQuantile to reject-list. And I think when using reject-list, we need a HashSet to record the classes that has been added to kvstore.
yeah, I agree it's optional to retain these CachedQuantile objects.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
Review comment:
fixed.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
If we don't write CachedQuantile to levelDB during the transition process, we might be able to get rid of reject list or allow list, and hence no need to handle some classes specially. I will test that idea and push the code changes later.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
agree
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
Review comment:
addressed
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
Review comment:
retrial logic is added.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
Explicit logic for handling particular classes are removed.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
Review comment:
we can update it to getStore().write(value)
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
+ override def onSwitchToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ store.setLevelDB(KVUtils.open(newStorePath, metadata))
+ memoryManager.release(appId, attempt.info.attemptId)
+ logInfo(s"Completely switched to LevelDB for app $appId / ${attempt.info.attemptId}.")
+ }
+
+ override def onSwitchToLevelDBFail(e: Exception): Unit = {
+ levelDB.close()
+ if (!isLeaseRolledBack.getAndSet(true)) {
+ lease.rollback()
+ }
Review comment:
I feel like this atomicBoolean variable is somewhat unnecessary. If the background thread can successfully start, the createHybridStore() will finish and return, hence no exceptions will be caught in the foreground thread. If exceptions occurred before backgroundThread.start(), the background thread won't be started and hence the lease can only be rollbacked on the foreground thread. What do you think? @HyukjinKwon @redsanket
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
+ override def onSwitchToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ store.setLevelDB(KVUtils.open(newStorePath, metadata))
+ memoryManager.release(appId, attempt.info.attemptId)
+ logInfo(s"Completely switched to LevelDB for app $appId / ${attempt.info.attemptId}.")
+ }
+
+ override def onSwitchToLevelDBFail(e: Exception): Unit = {
+ levelDB.close()
+ if (!isLeaseRolledBack.getAndSet(true)) {
+ lease.rollback()
+ }
Review comment:
I feel like this atomicBoolean variable is somewhat unnecessary. If the background thread can successfully start, the createHybridStore() will finish and return, hence no exceptions will be caught in the foreground thread. If exceptions occurred before backgroundThread.start(), the background thread won't be started and hence the lease can only be rollbacked on the foreground thread. What do you think? @HeartSaVioR @redsanket
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641951233
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123742/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420472617
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
+ }
+ })
+
+ rebuildAppStore(s, reader, attempt.info.lastUpdated.getTime())
+ s.stopBackgroundThreadAndSwitchToLevelDB()
+ store = s
+ } catch {
+ case _: IOException if !retried =>
+ // compaction may touch the file(s) which app rebuild wants to read
+ // compaction wouldn't run in short interval, so try again...
+ logWarning(s"Exception occurred while rebuilding app $appId - trying again...")
+ lease.rollback()
Review comment:
I missed the instance will be dereferenced, my bad.
It might be still ideal to clean up the instance explicitly, as the instance may keep the huge memory usages. Please note that this can happen in the middle of processing.
(The current logic may not be complete.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656819037
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-622212235
cc @vanzin FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641475758
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r451771981
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
Thanks for the information, I will try adding a `write(Iterator<E> values)` in this pr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644442574
**[Test build #124078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124078/testReport)** for PR 28412 at commit [`3412d29`](https://github.com/apache/spark/commit/3412d294a76a7348377f4d1a87327dc4fe510e96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-642054235
**[Test build #123770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123770/testReport)** for PR 28412 at commit [`76dbd18`](https://github.com/apache/spark/commit/76dbd184b659c2eeccba56f8201d470d939fceb4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654853105
So if I understand correctly, what we want to confirm is that the (size+compression)-memory ratio goes linearly on the number of tasks or not. Like short running interactive query (small number of tasks in overall) vs long running batch query (vs long running streaming query). I guess latter has been addressed in recent experiments, but former hasn't been experimented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r452055141
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
It is a function of how loaded your disk is, iops it can sustain, txn's leveldb can do concurrently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656819488
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-653246229
Let's try to overestimate for memory usage, as it's more critical than estimating disk usage (even for disk usage I think overestimating a bit is safer) and might lead to OOME if end users configure too tight on remaining heap memory.
zstd deserves to get at least 7x in the latest experiment, right? IMHO probably safer to apply 10x, and also apply 4x for other compression codecs as well. Let's hear @tgravescs voice on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638647662
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644070017
**[Test build #124053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124053/testReport)** for PR 28412 at commit [`76dbd18`](https://github.com/apache/spark/commit/76dbd184b659c2eeccba56f8201d470d939fceb4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621525704
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437600535
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
+ if (hybridStoreEnabled) {
+ try {
+ return createHybridStore(dm, appId, attempt, metadata)
+ } catch {
+ case e: Exception =>
+ logInfo(s"Failed to create hybrid store for $appId/${attempt.info.attemptId}." +
Review comment:
yeah, I think logInfo is ok since levelDB would work. And this usually occurs when the memory space is not enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624434103
Let's discuss first with the plan how to address the major concerns in comments, especially how to restrict the overall memory usage. I think that's a blocker for the production use.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r447727051
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
any update on the numbers?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440955307
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
I will add comments about this function and test compressed files later. So far I think this estimation is conservative and safe. For large uncompressed log files, the memory and disk usage are usually smaller than the estimated usage(half of the log file size). I measured some log files with size ~1g that I randomly picked from production-use spark jobs.
edit: The result seems not accurate because I am parsing spark2.4 log files using spark 3 history server.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641777321
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123722/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654866771
That said, probably we also haven't check how much hybrid KV store will help on the smaller app event log (like 100mb or even smaller in uncompressed). If that turns out to not help enough (either hybrid KV store only reduces a couple of seconds or replaying is fast enough without hybrid KV store), we can even exclude them on hybrid KV store path via having a lower bound (considering compression), as I assume the overhead of hybrid KV store is not that tiny.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-655128958
@tgravescs The latest number is based on uncompressed files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624441325
I'm not sure that's fairly simple to do. Concurrent load of applications can be happening in SHS, right? The default value of `spark.history.retainedApplications` is 50, which means maximum 50 apps can be loaded into cache at the same time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437837248
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1263,17 +1241,34 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
"trying again...")
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
retried = true
-
case e: Exception =>
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
throw e
}
}
+ // Create a LevelDB and start a background thread to dump data to LevelDB
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
Review comment:
do you mean exception thrown in the foreground thread or in the background thread? If it's in foreground thread, we can use a try-catch block to close the store when an exception is thrown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437837539
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1263,17 +1241,34 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
"trying again...")
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
retried = true
-
case e: Exception =>
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
throw e
}
}
+ // Create a LevelDB and start a background thread to dump data to LevelDB
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
Review comment:
Foreground, before switching method is called. Yeah I meant that. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r419768275
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
Review comment:
1. Here we should rollback the lease as well.
2. SHS would assume KVStore is loaded properly when the caller method returns. If I understand correctly, throwing exception here doesn't propagate to the caller method, which means HybridKVStore should still serve the content in any way, in-memory store for this case. (Caution of memory usage)
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
+ }
+ })
+
+ rebuildAppStore(s, reader, attempt.info.lastUpdated.getTime())
+ s.stopBackgroundThreadAndSwitchToLevelDB()
+ store = s
+ } catch {
+ case _: IOException if !retried =>
+ // compaction may touch the file(s) which app rebuild wants to read
+ // compaction wouldn't run in short interval, so try again...
+ logWarning(s"Exception occurred while rebuilding app $appId - trying again...")
+ lease.rollback()
Review comment:
`lease.rollback()` was enough to throw out intermediate LevelDB KVStore on temporary directory which fails to load at any reason. It doesn't look like the case of Hybrid KVStore.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
Review comment:
I've feeling that too much HybridKVStore implementation details are exposed here which could be abstracted away if we have proper interface for disk based loading-only KVStore for SHS.
E.g. suppose new interface on top of KVStore receives a Lease on `initialize()`, and exposes `commit()` & `rollback()` to handle the implementation details on each condition.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,9 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_KVSTORE_ENABLED = ConfigBuilder("spark.history.store.hybridKVStore.enabled")
+ .version("3.0.1")
Review comment:
1. This needs to have doc to describe the functionality, as well as proper caution on memory usage.
2. "3.1.0" is correct as of now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624478741
**[Test build #122335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122335/testReport)** for PR 28412 at commit [`141feed`](https://github.com/apache/spark/commit/141feed4a2537ff0481e4d03abad858df3540777).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641693298
cc. @vanzin @squito
Please take a look at this PR - looks like this is a notable improvement on reducing load latency in SHS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641689848
**[Test build #123719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123719/testReport)** for PR 28412 at commit [`1e514b9`](https://github.com/apache/spark/commit/1e514b910a56b719a08d6f7a7689a2a53dcc06a5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654478253
**[Test build #125136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125136/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641545654
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621525704
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638626777
**[Test build #123520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123520/testReport)** for PR 28412 at commit [`c6b5392`](https://github.com/apache/spark/commit/c6b539218465f94f82cd7562a99b8792640096e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624425131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang edited a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang edited a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641024447
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624424796
**[Test build #122333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122333/testReport)** for PR 28412 at commit [`34e4564`](https://github.com/apache/spark/commit/34e4564bcf17b1e9875b61445fb4af963af89449).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654455672
@tgravescs I measured the memory usage of an app that contains 4 jobs, and each job contains 1 stage, each stage contains 100000 tasks (the max retained tasks per stage by default). The number of entities stored in kvstore are 4 jobs, 4 stages, and 400000 tasks. The log size of this app is 1.1 GB, and the memory usage is 212 MB. I measured another app that contains 1000 jobs (the max retained jobs by default), each job contains 1 stage and each stage contains 400 tasks. The total number of entities for this app are 1000 jobs, 1000 stages, and 400000 tasks, which led to a 1.29 GB log file and 271 MB memory usage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656693014
**[Test build #125616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125616/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656158596
Yeah I’m OK either way. I think both ways wouldn’t bring issues in reality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r449140405
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
Hi, I updated the number on https://github.com/apache/spark/pull/28412#issuecomment-653110422.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420530840
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,9 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_KVSTORE_ENABLED = ConfigBuilder("spark.history.store.hybridKVStore.enabled")
+ .version("3.0.1")
Review comment:
Addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654561269
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638632450
**[Test build #123521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123521/testReport)** for PR 28412 at commit [`d710e46`](https://github.com/apache/spark/commit/d710e461d68d4eef87f55a7e930c81e51ae631b7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-622212235
cc @vanzin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654614118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658451540
Thanks! Merged into master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656693014
**[Test build #125616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125616/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641701635
**[Test build #123720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123720/testReport)** for PR 28412 at commit [`d6c7d98`](https://github.com/apache/spark/commit/d6c7d988bd9e39caebb9a33f8c01ee230b6c2a39).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624428013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r451892343
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
Hi @mridulm, I updated your code and used it on in-memory store - leveldb switching, but only saw little switching time improvement. I am not sure if somewhere wrong.
| log size, jobs and tasks per job | 2 jobs, 400 tasks per job | 10 jobs, 400 tasks per job | 50 jobs, 400 tasks per job | 100 jobs, 400 tasks per job | 200 jobs, 400 tasks per job | 500 jobs, 400 tasks per job | 1000 jobs, 400 tasks per job | 5 jobs, 100000 tasks per job |
| ------------------------------------------- | ------------------------- | -------------------------- | -------------------------- | --------------------------- | --------------------------- | --------------------------- | ---------------------------- | ---------------------------- |
| original switching time | 1s | 2s | 4s | 8s | 16s | 37s | 65s | 90s |
| switching time with write(Iterator<T> iter) | 1s | 1s | 4s | 7s | 13s | 34s | 58s | 84s |
The code:
```
for (klass <- klassMap.keys().asScala) {
val it = inMemoryStore.view(klass).closeableIterator()
levelDB.write(it)
}
```
```
public <T> void write(Iterator<T> iter) throws Exception {
Preconditions.checkArgument(iter != null, "Non-empty values required.");
List<T> values = new ArrayList<>();
iter.forEachRemaining(values::add);
// Group by class, in case there are values from different classes in the iterator
// Typical usecase is for this to be a single class.
for (Map.Entry<? extends Class<?>, ? extends List<?>> entry :
values.stream().collect(Collectors.groupingBy(Object::getClass)).entrySet()) {
final Iterator<?> valueIter = entry.getValue().iterator();
final Iterator<byte[]> serializedValueIter;
{
// deserialize outside synchronized block
List<byte[]> list = new ArrayList<>(entry.getValue().size());
for (Object value : entry.getValue()) {
list.add(serializer.serialize(value));
}
serializedValueIter = list.iterator();
}
final Class<?> valueClass = entry.getKey();
final LevelDBTypeInfo ti = getTypeInfo(valueClass);
// Batching updates per type
synchronized (ti) {
final LevelDBTypeInfo.Index naturalIndex = ti.naturalIndex();
final Collection<LevelDBTypeInfo.Index> indices = ti.indices();
try (WriteBatch batch = db().createWriteBatch()) {
while (valueIter.hasNext()) {
final Object value = valueIter.next();
assert serializedValueIter.hasNext();
final byte[] serializedObject = serializedValueIter.next();
Object existing;
try {
existing = get(naturalIndex.entityKey(null, value), valueClass);
} catch (NoSuchElementException e) {
existing = null;
}
PrefixCache cache = new PrefixCache(value);
byte[] naturalKey = naturalIndex.toKey(naturalIndex.getValue(value));
for (LevelDBTypeInfo.Index idx : indices) {
byte[] prefix = cache.getPrefix(idx);
idx.add(batch, value, existing, serializedObject, naturalKey, prefix);
}
}
assert !serializedValueIter.hasNext();
db().write(batch);
}
}
}
}
```
I think using multiple threads to write data to leveldb might shorten the switching time but it would introduce more overhead to SHS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621526046
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437598233
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
Review comment:
I would prefer to keep store.switchToLevelDB() in the try block and remove isLeaseRolledBack variable. If we move it out of the try block, we would need one more variable to track whether we have encountered exceptions during the rebuildAppStore(), otherwise, store.switchToLevelDB() will be executed when the rebuildAppStore() is failed due to IOException. What do you think?
I added a closed variable to indicate whether the hybrid store is closed, and not to start the background thread when the closed is true. This can address the issue above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437043628
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
This looks to be fragile. Is there a reason we have to apply allow-list instead of dumping all of the entities?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1264,6 +1335,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
endProcessing(rootPath)
}
}
+
Review comment:
nit: remove
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
This requires change for every time when any new api class is introduced, which is easy to miss - that's why I said 'fragile'. If we have to have some kind of allow-list vs reject-list, I think reject-list is better, because storing something after replaying is an exceptional case.
Actually I don't think they even need to be retained at all during transition - it's optional to migrate them to the level DB. All of them should be just some sort of cache. The status of KV store is stable after finished replaying, otherwise it would be problematic for in-progress application. Think about in-progress application - KV store becomes invalidated once the event log file is updated which means newly stored entries are lost. In practice it doesn't matter at all.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
Review comment:
This should be exhaustive, according to the build result. If we don't do anything for the None, please add `case None =>`
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
This requires change for every time when any new api class is introduced, which is easy to miss - that's why I said 'fragile'. If we have to have some kind of allow-list vs reject-list, I think reject-list is better, because storing something after replaying is really an exceptional case.
Actually I don't think they even need to be retained at all during transition - it's optional to migrate them to the level DB. All of them should be just some sort of cache. The status of KV store is stable after finished replaying, otherwise it would be problematic for in-progress application. Think about in-progress application - KV store becomes invalidated once the event log file is updated which means newly stored entries are lost. In practice it doesn't matter at all.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
If possible and works properly (though I guess so), it would be ideal if we don't specify the case "CachedQuantile" but don't apply all write operations to Level DB during the transition. It would trigger one more calculation/storing of CachedQuantiles after the transition to Level DB, but IMHO it wouldn't matter much as we already saved lots of seconds on loading and the overhead should be relatively small enough.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
If possible and works properly (though I guess so), it would be ideal if we don't specify the case "CachedQuantile" but don't apply all write operations to Level DB during the transition. It would trigger one more calculation/storing of CachedQuantiles after the transition to Level DB, but IMHO it wouldn't matter much as we already saved lots of seconds on loading and the overhead should be relatively small enough.
EDIT: sorry the idea was a bit sketched and I can imagine the possible issue during transition. I'm OK with specifying the case, but would still be ideal if we have better alternatives.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // Objects of these classes will be dumped to levelDB in the background thread
+ private var klassList: Seq[Class[_]] = List(
Review comment:
If possible and works properly, it would be ideal if we don't specify the case "CachedQuantile" but don't apply all write operations to Level DB during the transition. It would trigger one more calculation/storing of CachedQuantiles after the transition to Level DB, but IMHO it wouldn't matter much as we already saved lots of seconds on loading and the overhead should be relatively small enough.
EDIT: sorry the idea was a bit sketched and I can imagine the possible issue during transition. I'm OK with specifying the case, but would still be ideal if we have better alternatives.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
Review comment:
It may be worth to also describe the assumptions in the class doc that we don't expect write operations (except the case for caching) after calling switch to level DB.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
+ case Some(e) =>
+ listener.onSwitchToLevelDBFail(e)
Review comment:
This line can be moved to the catch statement.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
Review comment:
Can be `private`
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
Review comment:
Probably we may want to guarantee this to be executed once regardless of exception being thrown in join().
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
Review comment:
We can just remove this and remove the needs of `exception` variable, by applying below comments.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
+ case Some(e) =>
+ listener.onSwitchToLevelDBFail(e)
+ case None =>
+ listener.onSwitchToLevelDBSuccess()
Review comment:
And if we don't want to let background thread be crashed (because the exception may not be propagated anywhere) then we may need to guard `onXXX` and `inMemoryStore.close()` as well. If we just would like the background thread simply to be crashed, it'd be nice to clear out which behavior is expected on the rest of the threads (SHS behavior).
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
Review comment:
IMHO IllegalStateException seems to be a better fit, but we could also listen for other reviewers.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val MAX_IN_MEMORY_STORE_USAGE = ConfigBuilder("spark.history.store.hybridStore.maxMemoryUsage")
+ .version("3.1.0")
Review comment:
Probably better to mention it co-uses the heap memory so they should increase the heap memory in memory option for SHS if they enable hybrid store.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
+ case Some(e) =>
+ listener.onSwitchToLevelDBFail(e)
+ case None =>
+ listener.onSwitchToLevelDBSuccess()
Review comment:
This block can be just moved to the end of try statement.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
Review comment:
I guess this can be moved out of try statement - so that we can treat the error cases during replay and switch separately. Maybe this can also address your comment in below.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
Review comment:
ditto
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
+ case Some(e) =>
+ listener.onSwitchToLevelDBFail(e)
+ case None =>
+ listener.onSwitchToLevelDBSuccess()
+ shouldUseInMemoryStore.set(false)
+ inMemoryStore.close()
+ }
+ })
+ backgroundThread.setDaemon(true)
+ backgroundThread.setName("hybridstore-switch-to-leveldb")
+ backgroundThread.start()
+ }
+
+ /**
+ * This method return the store that we should use.
+ */
+ private def getStore(): KVStore = {
+ if (shouldUseInMemoryStore.get) {
+ inMemoryStore
+ } else {
+ levelDB
+ }
+ }
+
Review comment:
nit: I guess we don't prefer leaving empty line just for margin.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ } catch {
+ case ioe: IOException => throw ioe
+ } finally {
+ inMemoryStore.close()
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ backgroundThread = new Thread(() => {
+ var exception: Option[Exception] = None
+
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ } catch {
+ case e: Exception =>
+ exception = Some(e)
+ }
+
+ exception match {
+ case Some(e) =>
+ listener.onSwitchToLevelDBFail(e)
+ case None =>
+ listener.onSwitchToLevelDBSuccess()
+ shouldUseInMemoryStore.set(false)
+ inMemoryStore.close()
+ }
+ })
+ backgroundThread.setDaemon(true)
+ backgroundThread.setName("hybridstore-switch-to-leveldb")
+ backgroundThread.start()
+ }
+
+ /**
+ * This method return the store that we should use.
+ */
+ private def getStore(): KVStore = {
+ if (shouldUseInMemoryStore.get) {
+ inMemoryStore
+ } else {
+ levelDB
+ }
+ }
+
+}
+
+private[history] object HybridStore {
+
+ trait SwitchToLevelDBListener {
+
+ def onSwitchToLevelDBSuccess(): Unit
+
+ def onSwitchToLevelDBFail(e: Exception): Unit
+ }
+
Review comment:
nit: probably same here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang edited a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang edited a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654455672
@tgravescs I measured the memory usage of an app that contains 4 jobs, and each job contains 1 stage, each stage contains 100000 tasks (the max retained tasks per stage by default). The number of entities stored in kvstore are 4 jobs, 4 stages, and 400000 tasks. The log size of this app is 1.1 GB, and the memory usage is 212 MB. I measured another app that contains 1000 jobs (the max retained jobs by default), each job contains 1 stage and each stage contains 400 tasks. The total number of entities for this app are 1000 jobs, 1000 stages, and 400000 tasks, which led to a 1.29 GB log file and 271 MB memory usage.
The number of entities for the app in https://github.com/apache/spark/pull/28412#issuecomment-653110422 is 400 jobs, 400 stages, and 400000 tasks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-640924005
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621592228
**[Test build #122105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122105/testReport)** for PR 28412 at commit [`0164960`](https://github.com/apache/spark/commit/01649606321f3ce615fbf8fd06d6854a936593c1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644070471
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656594980
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654613562
**[Test build #125163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125163/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654479331
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641690214
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641545663
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123702/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656594552
**[Test build #125598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125598/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638632997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-639668009
I did a comparison between hybridstore, in-memory store, and levledb for parsing large log files. The environment is mac os, spark-3.1.0.
| store/ log file size | 250mb (SparkPi with 100000 iterations) | 1.2gb (SparkPi with 500000 iterations) | 2.3gb (SparkPi with 900000 iterations) |
| ---------------------- | ---------------------------------------- | ----------------------------------------- | ----------------------------------------- |
| HybridStore | 8s to parse, 16s to dump data to leveldb | 33s to parse, 16s to dump data to leveldb | 68s to parse, 15s to dump data to leveldb |
| InMemoryStore | 8s to parse | 34s to parse | 68s to parse |
| LevelDB | 28s to parse | 187s to parse | 434s to parse |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658211580
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641701924
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440858998
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ listener.onSwitchToLevelDBSuccess()
+ shouldUseInMemoryStore.set(false)
Review comment:
it doesn't now, but if someone were to modify it in future it could.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440489736
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -128,6 +128,10 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
private val storePath = conf.get(LOCAL_STORE_DIR).map(new File(_))
private val fastInProgressParsing = conf.get(FAST_IN_PROGRESS_PARSING)
+ private val hybridStoreEnabled = conf.get(History.HYBRID_STORE_ENABLED)
+
+ private val memoryManager = new HistoryServerMemoryManager(conf)
Review comment:
Addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656690866
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r436151064
##########
File path: common/kvstore/src/test/java/org/apache/spark/util/kvstore/HybridStoreSuite.java
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.NoSuchElementException;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.junit.Assert.*;
+
+public class HybridStoreSuite {
+ private LevelDB db;
+ private File dbpath;
+
+ @After
+ public void cleanup() throws Exception {
+ if (db != null) {
+ db.close();
+ }
+ if (dbpath != null) {
+ FileUtils.deleteQuietly(dbpath);
+ }
+ }
+
+ @Before
+ public void setup() throws Exception {
+ dbpath = File.createTempFile("test.", ".ldb");
+ dbpath.delete();
+ db = new LevelDB(dbpath);
+ }
+
+ @Test
+ public void testMultipleObjectWriteReadDelete() throws Exception {
+ HybridStore store = createHybridStore();
+
+ CustomType1 t1 = createCustomType1(1);
+ CustomType1 t2 = createCustomType1(2);
+
+ try {
+ store.read(t1.getClass(), t1.key);
+ fail("Expected exception for non-existent object.");
+ } catch (NoSuchElementException nsee) {
+ // Expected.
+ }
+
+ store.write(t1);
+ store.write(t2);
+
+ assertEquals(t1, store.read(t1.getClass(), t1.key));
+ assertEquals(t2, store.read(t2.getClass(), t2.key));
+ assertEquals(2L, store.count(t1.getClass()));
+
+ store.delete(t2.getClass(), t2.key);
+ assertEquals(1L, store.count(t1.getClass()));
+
+ switchToLevelDB(store);
+ assertEquals(t1, store.read(t1.getClass(), t1.key));
+ try {
+ store.read(t2.getClass(), t2.key);
+ fail("Expected exception for deleted object.");
+ } catch (NoSuchElementException nsee) {
+ // Expected.
+ }
+ }
+
+ @Test
+ public void testUpdate() throws Exception {
+ HybridStore store = createHybridStore();
+
+ CustomType1 t = createCustomType1(1);
+
+ store.write(t);
+
+ t.name = "anotherName";
+
+ store.write(t);
+
+ switchToLevelDB(store);
+ assertEquals(1, store.count(t.getClass()));
+ assertEquals("anotherName", store.read(t.getClass(), t.key).name);
+ }
+
+ @Test
+ public void testArrayIndices() throws Exception {
+ HybridStore store = createHybridStore();
+
+ ArrayKeyIndexType o = new ArrayKeyIndexType();
+ o.key = new int[] { 1, 2 };
+ o.id = new String[] { "3", "4" };
+
+ store.write(o);
+
+ switchToLevelDB(store);
+ assertEquals(o, store.read(ArrayKeyIndexType.class, o.key));
+ assertEquals(o, store.view(ArrayKeyIndexType.class).index("id").first(o.id).iterator().next());
+ }
+
+ @Test
+ public void testBasicIteration() throws Exception {
+ HybridStore store = createHybridStore();
+
+ CustomType1 t1 = createCustomType1(1);
+ store.write(t1);
+
+ CustomType1 t2 = createCustomType1(2);
+ store.write(t2);
+
+ switchToLevelDB(store);
+ assertEquals(t1.id, store.view(t1.getClass()).iterator().next().id);
+ assertEquals(t2.id, store.view(t1.getClass()).skip(1).iterator().next().id);
+ assertEquals(t2.id, store.view(t1.getClass()).skip(1).max(1).iterator().next().id);
+ assertEquals(t1.id,
+ store.view(t1.getClass()).first(t1.key).max(1).iterator().next().id);
+ assertEquals(t2.id,
+ store.view(t1.getClass()).first(t2.key).max(1).iterator().next().id);
+ }
+
+ @Test
+ public void testSkip() throws Exception {
+ HybridStore store = createHybridStore();
+
+ for (int i = 0; i < 10; i++) {
+ store.write(createCustomType1(i));
+ }
+
+ switchToLevelDB(store);
+ KVStoreIterator<CustomType1> it = store.view(CustomType1.class).closeableIterator();
+ assertTrue(it.hasNext());
+ assertTrue(it.skip(5));
+ assertEquals("key5", it.next().key);
+ assertTrue(it.skip(3));
+ assertEquals("key9", it.next().key);
+ assertFalse(it.hasNext());
+ }
+
+ @Test
+ public void testRejectWriting() throws Exception {
+ HybridStore store = createHybridStore();
+ switchToLevelDB(store);
+ try {
+ store.write(createCustomType1(1));
+ fail("Expected exception for writing object after switching to LevelDB");
+ } catch (RuntimeException re) {
+ // Expected.
+ }
+ }
+
+ private HybridStore createHybridStore() {
+ HybridStore store = new HybridStore();
+ store.setLevelDB(db);
+ return store;
+ }
+
+ private void switchToLevelDB(HybridStore store) throws Exception {
+ SwitchingListener listener = new SwitchingListener();
+ store.switchingToLevelDB(listener);
+ assert(listener.waitUntilDone());
+ }
+
+ private class SwitchingListener implements HybridStore.SwitchingToLevelDBListener {
+ private LinkedBlockingQueue<String> results = new LinkedBlockingQueue<>();
+
+ @Override
+ public void onSwitchingToLevelDBSuccess() {
+ try {
+ results.put("Succeed");
Review comment:
Can Succeed, Failed be enums instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420525489
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
+ }
+ })
+
+ rebuildAppStore(s, reader, attempt.info.lastUpdated.getTime())
+ s.stopBackgroundThreadAndSwitchToLevelDB()
+ store = s
+ } catch {
+ case _: IOException if !retried =>
+ // compaction may touch the file(s) which app rebuild wants to read
+ // compaction wouldn't run in short interval, so try again...
+ logWarning(s"Exception occurred while rebuilding app $appId - trying again...")
+ lease.rollback()
Review comment:
Hybrid kvstore is now cleaned up explicitly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-636907866
**[Test build #123375 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123375/testReport)** for PR 28412 at commit [`a796231`](https://github.com/apache/spark/commit/a796231c82ca5b2206b25d4bf6bea951ea3ee631).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654835055
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r451795960
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
If this helps, this is what I had written up for level db - for memory store, the default list traversal + write is good enough :
```
@Override
public void write(List<?> values) throws Exception {
Preconditions.checkArgument(values != null && !values.isEmpty(),
"Non-empty values required.");
// Group by class, in case there are values from different classes in the iterator
// Typical usecase is for this to be a single class.
for (Map.Entry<? extends Class<?>, ? extends List<?>> entry :
values.stream().collect(Collectors.groupingBy(Object::getClass)).entrySet()) {
final Iterator<?> valueIter = entry.getValue().iterator();
final Iterator<byte[]> serializedValueIter;
{
// deserialize outside synchronized block
List<byte[]> list = new ArrayList<>(entry.getValue().size());
for (Object value : entry.getValue()) {
list.add(serializer.serialize(value));
}
serializedValueIter = list.iterator();
}
final Class<?> valueClass = entry.getKey();
final LevelDBTypeInfo ti = getTypeInfo(valueClass);
// Batching updates per type
synchronized (ti) {
final LevelDBTypeInfo.Index naturalIndex = ti.naturalIndex();
final Collection<LevelDBTypeInfo.Index> indices = ti.indices();
try (WriteBatch batch = db().createWriteBatch()) {
while (valueIter.hasNext()) {
final Object value = valueIter.next();
assert serializedValueIter.hasNext();
final byte[] serializedObject = serializedValueIter.next();
Object existing;
try {
existing = get(naturalIndex.entityKey(null, value), valueClass);
} catch (NoSuchElementException e) {
existing = null;
}
PrefixCache cache = new PrefixCache(value);
byte[] naturalKey = naturalIndex.toKey(naturalIndex.getValue(value));
for (LevelDBTypeInfo.Index idx : indices) {
byte[] prefix = cache.getPrefix(idx);
idx.add(batch, value, existing, serializedObject, naturalKey, prefix);
}
}
assert !serializedValueIter.hasNext();
db().write(batch);
}
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621536233
@jiangxb1987 from my testing result, loading 1g file, HybridKVStore takes 23s to parse (that means, users only need to wait for 23s to see the UI), LevelDB takes 69s.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621630288
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654614118
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656909607
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654562198
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654561949
**[Test build #125163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125163/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641545201
**[Test build #123702 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123702/testReport)** for PR 28412 at commit [`3494307`](https://github.com/apache/spark/commit/3494307edea454bf37008f4db1c23ee16263b871).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654479331
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-655002716
I measured the memory usage of some smaller apps, the results are:
* 200 jobs, 400 tasks for each job: 265 MB file size, 57.9 MB memory usage.
* 100 jobs, 400 tasks for each job: 133 MB file size, 28.5 MB memory usage.
* 50 jobs, 400 tasks for each job: 67 MB file size, 14.9 MB memory usage.
* 20 jobs: 400 tasks for each job: 35 MB file size, 8.3 MB memory usage
-----------------------------------------------------------------------------------------------
* 10 jobs: 400 tasks for each job: 15 MB file size, 4.7 MB memory usage.
* 1 job, 400 tasks for each job: 3.7 MB file size, 2.2 MB memory usage.
* 1 job, 40 tasks for each job: 512 KB file size, 727 KB memory usage.
I found that the ratio of memory_usage / filesize is stable at ¼ for log files larger than 30 MB. For log files with size less than 15 MB, the ratio of memory_usage/ filesize is greater than ¼, and the ratio increases as the file size decrease.
The difference in serving latency between in-memory and leveldb store depends on the machine's performance. But personally I feel parsing a log file greater than 50MB with the hybrid store can have a notable improvement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624424796
**[Test build #122333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122333/testReport)** for PR 28412 at commit [`34e4564`](https://github.com/apache/spark/commit/34e4564bcf17b1e9875b61445fb4af963af89449).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440225765
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(listener: HybridStore.SwitchToLevelDBListener): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
+ }
+ listener.onSwitchToLevelDBSuccess()
+ shouldUseInMemoryStore.set(false)
Review comment:
do we need to reset this to true if the exception is thrown in close?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -128,6 +128,10 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
private val storePath = conf.get(LOCAL_STORE_DIR).map(new File(_))
private val fastInProgressParsing = conf.get(FAST_IN_PROGRESS_PARSING)
+ private val hybridStoreEnabled = conf.get(History.HYBRID_STORE_ENABLED)
+
+ private val memoryManager = new HistoryServerMemoryManager(conf)
Review comment:
memory manager only appears to be used with hybrid store, so we shouldn't create and initialize it if not using the hybrid store
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,20 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ val MAX_IN_MEMORY_STORE_USAGE = ConfigBuilder("spark.history.store.hybridStore.maxMemoryUsage")
Review comment:
you need to update monitoring.md doc
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
Review comment:
I would extend the comment to say try it first, fall back to leveldb store.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
how accurate is this in practice? Will depend on the codec chosen. did you try some large files?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
+ if (hybridStoreEnabled) {
+ try {
+ return createHybridStore(dm, appId, attempt, metadata)
+ } catch {
+ case e: Exception =>
+ logInfo(s"Failed to create HybridStore for $appId/${attempt.info.attemptId}." +
+ " Using LevelDB.", e)
+ }
+ }
+
Review comment:
I find this function hard to read now. I would say at least put a comment here and possibly split the below logic into a separate function for easier readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #28412:
URL: https://github.com/apache/spark/pull/28412
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641475152
**[Test build #123702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123702/testReport)** for PR 28412 at commit [`3494307`](https://github.com/apache/spark/commit/3494307edea454bf37008f4db1c23ee16263b871).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641803852
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654838383
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656443921
**[Test build #125522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125522/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656390917
**[Test build #125522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125522/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437784544
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,78 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ var retried = false
+ var hybridStore: HybridStore = null
+ while (hybridStore == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ // Throws an exception if the memory space is not enough
+ memoryManager.lease(appId, attempt.info.attemptId, reader.totalSize, isCompressed)
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ rebuildAppStore(store, reader, attempt.info.lastUpdated.getTime())
+
+ // Start the background thread to dump data to levelDB when writing to
+ // InMemoryStore is completed.
+ store.switchToLevelDB(new HybridStore.SwitchToLevelDBListener {
Review comment:
The reason to split is that retrial logic is only valid for replaying - that said, you don't need to prepare Level DB and deal with disk lease (acquire/rollback) while replaying. You just need to deal with it "after" replaying which should simplify the logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641710484
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437780461
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
+ if (hybridStoreEnabled) {
+ try {
+ return createHybridStore(dm, appId, attempt, metadata)
+ } catch {
+ case e: Exception =>
+ logInfo(s"Failed to create hybrid store for $appId/${attempt.info.attemptId}." +
Review comment:
warn seems to be appropriate here however
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621660613
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-655129919
could you run just a couple of the same test with Zstd ( you don't need to run all those combinations)?
Note thanks again for running all those, I know it takes time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654856475
that is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437833266
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1263,17 +1241,34 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
"trying again...")
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
retried = true
-
case e: Exception =>
store.close()
memoryManager.release(appId, attempt.info.attemptId)
- lease.rollback()
throw e
}
}
+ // Create a LevelDB and start a background thread to dump data to LevelDB
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
Review comment:
There are still chances here exceptions have been thrown, and then we’ll leak HybridStore instance with memory being allocated, and never be released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624907362
@tgravescs What I saw in FsHistoryProvider is that "spark.history.fs.numReplayThreads" is used to create a thread pool to mergeApplicationListing and compact. These threads seem are not responsible for parsing a complete single event. But I am not sure if my understanding is correct. And In hybrid kvstore, each instance has one writing thread to write data to leveldb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-621592228
**[Test build #122105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122105/testReport)** for PR 28412 at commit [`0164960`](https://github.com/apache/spark/commit/01649606321f3ce615fbf8fd06d6854a936593c1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420524061
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
Review comment:
I found it's difficult to pass Lease to hybrid kvstore, since Lease class is only visible within the History package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658315222
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654853105
So if I understand correctly, what we want to confirm is that the (size+compression):memory ratio goes linearly on the number of tasks or not. Like short running interactive query (small number of tasks in overall) vs long running batch query (vs long running streaming query). I guess latter has been addressed in recent experiments, but former hasn't been experimented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641764410
**[Test build #123720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123720/testReport)** for PR 28412 at commit [`d6c7d98`](https://github.com/apache/spark/commit/d6c7d988bd9e39caebb9a33f8c01ee230b6c2a39).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624429091
**[Test build #122335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122335/testReport)** for PR 28412 at commit [`141feed`](https://github.com/apache/spark/commit/141feed4a2537ff0481e4d03abad858df3540777).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641732806
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656819488
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644429902
**[Test build #124077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124077/testReport)** for PR 28412 at commit [`025b13c`](https://github.com/apache/spark/commit/025b13c5d51fe15333c520b586edacb381af6bf7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-638515964
> Thanks for the contribution. The concept looks great.
>
> I've skimmed the implementation, and got the feeling that this is a bit complicated because of dual writes on the foreground & background. The complication is even exposed to the FsHistoryProvider.
>
> Given the HybridKVStore is only used for loading event log and no further exposed for modification (AFAIK - please correct me if I'm missing here), I'm seeing a chance to simplify the logic - dump in-memory KVStore back to LevelDB (in background) once the loading is done to the in-memory KVStore. (KVStore should be read-only and reject the writes.) We won't need to deal with dual writes on concurrent threads, only need to switch the KVStore correctly.
>
> This wouldn't bring latency on serving content - needs more seconds to write to LevelDB, in other words, needs more seconds to keep up the memory usage. (Not sure how long it would be.)
>
> What do you think?
@HyukjinKwon KVStore is not read-only after the rebuildAppStore() is done. In core/src/main/scala/org/apache/spark/status/AppStatusStore.scala#357, kvstore writes CachedQuantile. AFAIK, this is the only type of objects that can be written to kvstore after the loading is done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r436233082
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
Review comment:
Same here.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
Review comment:
Same here.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -128,6 +129,10 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
private val storePath = conf.get(LOCAL_STORE_DIR).map(new File(_))
private val fastInProgressParsing = conf.get(FAST_IN_PROGRESS_PARSING)
+ private val hybridStoreEnabled = conf.get(History.HYBRID_STORE_ENABLED)
+ private val maxInMemoryStoreUsage = conf.get(History.MAX_IN_MEMORY_STORE_USAGE)
+ private val currentInMemoryStoreUsage = new java.util.concurrent.atomic.AtomicLong(0L)
Review comment:
Let's have a separate class for memory lease, like HistoryServerDiskManager. This would be pretty much simpler than HistoryServerDiskManager, but avoid dealing with memory calculation in FsHistoryProvider.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
Review comment:
This is the only the kind being stored after replaying, though I think not migrating them is OK, because they can be recomputed. Trade-off between complexity vs possible overhead to recompute if it's being computed once during dumping to level DB.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
Review comment:
Same here.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
Review comment:
Let's make it shorter via `return getStore().getMetadata(klass);`. We don't do any kind of null check hence no need to let them separate.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
Review comment:
Same here.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
Review comment:
Same here.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
Review comment:
Same here.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
Review comment:
Same here.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type, index, indexedValue);
+ return count;
+ }
+
+ @Override
+ public void close() throws IOException {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join();
+ }
+ if (levelDB != null) {
+ levelDB.close();
+ }
+ if (inMemoryStore != null) {
+ inMemoryStore.close();
+ }
+ } catch (Exception e) {
+ if (e instanceof IOException) {
+ throw (IOException) e;
+ }
+ }
+ }
+
+ @Override
+ public <T> boolean removeAllByIndexValues(
+ Class<T> klass,
+ String index,
+ Collection<?> indexValues) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
Review comment:
Same here, if the line doesn't exceed the limit.
##########
File path: common/kvstore/src/test/java/org/apache/spark/util/kvstore/HybridStoreSuite.java
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.NoSuchElementException;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.junit.Assert.*;
+
+public class HybridStoreSuite {
+ private LevelDB db;
+ private File dbpath;
+
+ @After
+ public void cleanup() throws Exception {
+ if (db != null) {
+ db.close();
+ }
+ if (dbpath != null) {
+ FileUtils.deleteQuietly(dbpath);
+ }
+ }
+
+ @Before
+ public void setup() throws Exception {
+ dbpath = File.createTempFile("test.", ".ldb");
+ dbpath.delete();
+ db = new LevelDB(dbpath);
+ }
+
+ @Test
+ public void testMultipleObjectWriteReadDelete() throws Exception {
+ HybridStore store = createHybridStore();
+
+ CustomType1 t1 = createCustomType1(1);
+ CustomType1 t2 = createCustomType1(2);
+
+ try {
+ store.read(t1.getClass(), t1.key);
+ fail("Expected exception for non-existent object.");
+ } catch (NoSuchElementException nsee) {
+ // Expected.
+ }
+
+ store.write(t1);
+ store.write(t2);
+
+ assertEquals(t1, store.read(t1.getClass(), t1.key));
+ assertEquals(t2, store.read(t2.getClass(), t2.key));
+ assertEquals(2L, store.count(t1.getClass()));
+
+ store.delete(t2.getClass(), t2.key);
+ assertEquals(1L, store.count(t1.getClass()));
+
+ switchToLevelDB(store);
+ assertEquals(t1, store.read(t1.getClass(), t1.key));
+ try {
+ store.read(t2.getClass(), t2.key);
+ fail("Expected exception for deleted object.");
+ } catch (NoSuchElementException nsee) {
+ // Expected.
+ }
+ }
+
+ @Test
+ public void testUpdate() throws Exception {
+ HybridStore store = createHybridStore();
+
+ CustomType1 t = createCustomType1(1);
+
+ store.write(t);
+
+ t.name = "anotherName";
+
+ store.write(t);
+
+ switchToLevelDB(store);
+ assertEquals(1, store.count(t.getClass()));
+ assertEquals("anotherName", store.read(t.getClass(), t.key).name);
+ }
+
+ @Test
+ public void testArrayIndices() throws Exception {
+ HybridStore store = createHybridStore();
+
+ ArrayKeyIndexType o = new ArrayKeyIndexType();
+ o.key = new int[] { 1, 2 };
+ o.id = new String[] { "3", "4" };
+
+ store.write(o);
+
+ switchToLevelDB(store);
+ assertEquals(o, store.read(ArrayKeyIndexType.class, o.key));
+ assertEquals(o, store.view(ArrayKeyIndexType.class).index("id").first(o.id).iterator().next());
+ }
+
+ @Test
+ public void testBasicIteration() throws Exception {
+ HybridStore store = createHybridStore();
+
+ CustomType1 t1 = createCustomType1(1);
+ store.write(t1);
+
+ CustomType1 t2 = createCustomType1(2);
+ store.write(t2);
+
+ switchToLevelDB(store);
+ assertEquals(t1.id, store.view(t1.getClass()).iterator().next().id);
+ assertEquals(t2.id, store.view(t1.getClass()).skip(1).iterator().next().id);
+ assertEquals(t2.id, store.view(t1.getClass()).skip(1).max(1).iterator().next().id);
+ assertEquals(t1.id,
+ store.view(t1.getClass()).first(t1.key).max(1).iterator().next().id);
+ assertEquals(t2.id,
+ store.view(t1.getClass()).first(t2.key).max(1).iterator().next().id);
+ }
+
+ @Test
+ public void testSkip() throws Exception {
+ HybridStore store = createHybridStore();
+
+ for (int i = 0; i < 10; i++) {
+ store.write(createCustomType1(i));
+ }
+
+ switchToLevelDB(store);
+ KVStoreIterator<CustomType1> it = store.view(CustomType1.class).closeableIterator();
+ assertTrue(it.hasNext());
+ assertTrue(it.skip(5));
+ assertEquals("key5", it.next().key);
+ assertTrue(it.skip(3));
+ assertEquals("key9", it.next().key);
+ assertFalse(it.hasNext());
+ }
+
+ @Test
+ public void testRejectWriting() throws Exception {
+ HybridStore store = createHybridStore();
+ switchToLevelDB(store);
+ try {
+ store.write(createCustomType1(1));
+ fail("Expected exception for writing object after switching to LevelDB");
+ } catch (RuntimeException re) {
+ // Expected.
+ }
+ }
+
+ private HybridStore createHybridStore() {
+ HybridStore store = new HybridStore();
+ store.setLevelDB(db);
+ return store;
+ }
+
+ private void switchToLevelDB(HybridStore store) throws Exception {
+ SwitchingListener listener = new SwitchingListener();
+ store.switchingToLevelDB(listener);
+ assert(listener.waitUntilDone());
+ }
+
+ private class SwitchingListener implements HybridStore.SwitchingToLevelDBListener {
+ private LinkedBlockingQueue<String> results = new LinkedBlockingQueue<>();
+
+ @Override
+ public void onSwitchingToLevelDBSuccess() {
+ try {
+ results.put("Succeed");
Review comment:
+1 on the suggestion.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type, index, indexedValue);
+ return count;
+ }
+
+ @Override
+ public void close() throws IOException {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join();
+ }
+ if (levelDB != null) {
+ levelDB.close();
+ }
+ if (inMemoryStore != null) {
+ inMemoryStore.close();
+ }
+ } catch (Exception e) {
+ if (e instanceof IOException) {
+ throw (IOException) e;
+ }
+ }
+ }
+
+ @Override
+ public <T> boolean removeAllByIndexValues(
+ Class<T> klass,
+ String index,
+ Collection<?> indexValues) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ boolean removed = store.removeAllByIndexValues(klass, index, indexValues);
+ return removed;
+ }
+
+ public void setLevelDB(LevelDB levelDB) {
+ this.levelDB = levelDB;
+ }
+
+ public void setCachedQuantileKlass(Class<?> klass) {
+ this.cachedQuantileKlass = klass;
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ public void switchingToLevelDB(SwitchingToLevelDBListener listener) throws Exception {
+ // A background thread that dumps data to levelDB
+ backgroundThread = new Thread(new Runnable() {
+ public void run() {
+ Exception exceptionCaught = null;
+
+ try {
+ for (Class<?> klass : klassMap.keySet()) {
+ KVStoreIterator<?> it = inMemoryStore.view(klass).closeableIterator();
+ while (it.hasNext()) {
+ levelDB.write(it.next());
+ }
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
Review comment:
Please don't print on the sysout/syserr - if the exception is being logged by somewhere then it's fine not to print here.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val MAX_IN_MEMORY_STORE_USAGE = ConfigBuilder("spark.history.store.hybridStore.maxMemoryUsage")
+ .version("3.1.0")
Review comment:
Please add doc for this configuration as well, with specifying default value.
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/HybridStore.java
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import org.apache.spark.annotation.Private;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.Collection;
+
+/**
+ * Implementation of KVStore that writes data to InMemoryStore at first and uses
+ * a background thread to dump data to LevelDB once the writing to InMemoryStore
+ * is completed.
+ */
+@Private
+public class HybridStore implements KVStore {
+
+ private InMemoryStore inMemoryStore = new InMemoryStore();
+ private LevelDB levelDB = null;
+
+ // Flag to indicate if we should use inMemoryStore Or levelDB.
+ private AtomicBoolean shouldUseInMemoryStore = new AtomicBoolean(true);
+
+ // A background thread that dumps data in inMemoryStore to levelDB
+ private Thread backgroundThread = null;
+
+ // A hash map that stores all class types (except CachedQuantile) that had been writen
+ // to inMemoryStore.
+ private ConcurrentHashMap<Class<?>, Boolean> klassMap = new ConcurrentHashMap<>();
+
+ // CachedQuantile can be written to kvstore after rebuildAppStore(), so we need
+ // to handle it specially to avoid conflicts. We will use a queue store CachedQuantile
+ // objects when the underlying store is inMemoryStore, and dump these objects to levelDB
+ // before the switch completes.
+ private Class<?> cachedQuantileKlass = null;
+ private ConcurrentLinkedQueue<Object> cachedQuantileQueue = new ConcurrentLinkedQueue<>();
+
+
+ @Override
+ public <T> T getMetadata(Class<T> klass) throws Exception {
+ KVStore store = getStore();
+ T metaData = store.getMetadata(klass);
+ return metaData;
+ }
+
+ @Override
+ public void setMetadata(Object value) throws Exception {
+ KVStore store = getStore();
+ store.setMetadata(value);
+ }
+
+ @Override
+ public <T> T read(Class<T> klass, Object naturalKey) throws Exception {
+ KVStore store = getStore();
+ T value = store.read(klass, naturalKey);
+ return value;
+ }
+
+ @Override
+ public void write(Object value) throws Exception {
+ Class<?> klass = value.getClass();
+
+ if (backgroundThread != null && !klass.equals(cachedQuantileKlass)) {
+ throw new RuntimeException("write() for objects other than CachedQuantile " +
+ "shouldn't be called after the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.write(value);
+
+ if (store instanceof InMemoryStore) {
+ if (klass.equals(cachedQuantileKlass)) {
+ cachedQuantileQueue.add(value);
+ } else {
+ klassMap.putIfAbsent(klass, true);
+ }
+ }
+ }
+
+ @Override
+ public void delete(Class<?> type, Object naturalKey) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ store.delete(type, naturalKey);
+ }
+
+ @Override
+ public <T> KVStoreView<T> view(Class<T> type) throws Exception {
+ KVStore store = getStore();
+ KVStoreView<T> view = store.view(type);
+ return view;
+ }
+
+ @Override
+ public long count(Class<?> type) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type);
+ return count;
+ }
+
+ @Override
+ public long count(Class<?> type, String index, Object indexedValue) throws Exception {
+ KVStore store = getStore();
+ long count = store.count(type, index, indexedValue);
+ return count;
+ }
+
+ @Override
+ public void close() throws IOException {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join();
+ }
+ if (levelDB != null) {
+ levelDB.close();
+ }
+ if (inMemoryStore != null) {
+ inMemoryStore.close();
+ }
+ } catch (Exception e) {
+ if (e instanceof IOException) {
+ throw (IOException) e;
+ }
+ }
+ }
+
+ @Override
+ public <T> boolean removeAllByIndexValues(
+ Class<T> klass,
+ String index,
+ Collection<?> indexValues) throws Exception {
+ if (backgroundThread != null) {
+ throw new RuntimeException("removeAllByIndexValues() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB");
+ }
+
+ KVStore store = getStore();
+ boolean removed = store.removeAllByIndexValues(klass, index, indexValues);
+ return removed;
+ }
+
+ public void setLevelDB(LevelDB levelDB) {
+ this.levelDB = levelDB;
+ }
+
+ public void setCachedQuantileKlass(Class<?> klass) {
+ this.cachedQuantileKlass = klass;
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ public void switchingToLevelDB(SwitchingToLevelDBListener listener) throws Exception {
+ // A background thread that dumps data to levelDB
+ backgroundThread = new Thread(new Runnable() {
+ public void run() {
+ Exception exceptionCaught = null;
+
+ try {
+ for (Class<?> klass : klassMap.keySet()) {
+ KVStoreIterator<?> it = inMemoryStore.view(klass).closeableIterator();
+ while (it.hasNext()) {
+ levelDB.write(it.next());
+ }
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ exceptionCaught = e;
+ }
+
+ if (exceptionCaught == null) {
+ // Switch to levelDB and close inMemoryStore
+ shouldUseInMemoryStore.set(false);
+
+ // Dump CachedQuantile objects to levelDB
+ try {
+ while(cachedQuantileQueue.size() > 0) {
+ levelDB.write(cachedQuantileQueue.poll());
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
Review comment:
Same here. And isn't the exception part of failure on switching, which shouldn't be swallowed?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1172,17 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // If hybrid store is enabled, try it first.
Review comment:
The retrial below is due to the compaction operation being held concurrently. HybridStore should also follow the retrial logic as well.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
Review comment:
I think it doesn't need to be a kind of caution, but it should mention the below configuration and how it affects the usage of the heap memory.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
+ .version("3.1.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
Please set it false by default, as it consumes more memory (max 2g) and end users may not be noticed by the change and encounter OOM.
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1197,6 +1213,71 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
KVUtils.open(newStorePath, metadata)
}
+ private def createHybridStore(
+ dm: HistoryServerDiskManager,
+ appId: String,
+ attempt: AttemptInfoWrapper,
+ metadata: AppStatusStoreMetadata): KVStore = {
+
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+
+ val memoryUsage = approximateMemoryUsage(reader.totalSize, isCompressed)
+ if (currentInMemoryStoreUsage.get + memoryUsage > maxInMemoryStoreUsage) {
+ throw new IllegalStateException("Not enough in-memory storage to create hybrid store.")
+ }
+ currentInMemoryStoreUsage.addAndGet(memoryUsage)
+ logInfo(s"Attempt creating hybrid store to parse $appId / ${attempt.info.attemptId}. " +
+ s"Requested ${Utils.bytesToString(memoryUsage)} in-memory storage quota.")
+
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ val isLeaseRolledBack = new java.util.concurrent.atomic.AtomicBoolean(false)
+ var store: HybridStore = null
+ try {
+ store = new HybridStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ store.setLevelDB(levelDB)
+ store.setCachedQuantileKlass(classOf[CachedQuantile])
Review comment:
Since HybridStore depends on the logic in AppStatusStore (hence specific to SHS), I think it's not required for HybridStore to be placed in common-kvstore module. HybridStore can be moved to core module (this kind of workaround is not needed then) and there's another chance you can simplify the code (+ test code) via implementing to Scala.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644429902
**[Test build #124077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124077/testReport)** for PR 28412 at commit [`025b13c`](https://github.com/apache/spark/commit/025b13c5d51fe15333c520b586edacb381af6bf7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654532523
**[Test build #125136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125136/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654533150
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437788564
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new RuntimeException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ if (levelDB != null) {
Review comment:
Isn't join() able to throw InterruptedException? That's not a runtime exception but you're playing with Scala which ignores checked exception so you still need to be careful.
And IMHO, generally at any case we must ensure both level DB store and in memory store are closed because that's a resource leak.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r440907281
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.History._
+import org.apache.spark.util.Utils
+
+/**
+ * A class used to keep track of in-memory store usage by the SHS.
+ */
+private class HistoryServerMemoryManager(
+ conf: SparkConf) extends Logging {
+
+ private val maxUsage = conf.get(MAX_IN_MEMORY_STORE_USAGE)
+ private val currentUsage = new AtomicLong(0L)
+ private val active = new HashMap[(String, Option[String]), Long]()
+
+ def initialize(): Unit = {
+ logInfo("Initialized memory manager: " +
+ s"current usage = ${Utils.bytesToString(currentUsage.get())}, " +
+ s"max usage = ${Utils.bytesToString(maxUsage)}")
+ }
+
+ def lease(
+ appId: String,
+ attemptId: Option[String],
+ eventLogSize: Long,
+ isCompressed: Boolean): Unit = {
+ val memoryUsage = approximateMemoryUsage(eventLogSize, isCompressed)
+ if (memoryUsage + currentUsage.get > maxUsage) {
+ throw new RuntimeException("Not enough memory to create hybrid store " +
+ s"for app $appId / $attemptId.")
+ }
+ active.synchronized {
+ active(appId -> attemptId) = memoryUsage
+ }
+ currentUsage.addAndGet(memoryUsage)
+ logInfo(s"Leasing ${Utils.bytesToString(memoryUsage)} memory usage for " +
+ s"app $appId / $attemptId")
+ }
+
+ def release(appId: String, attemptId: Option[String]): Unit = {
+ val memoryUsage = active.synchronized { active.remove(appId -> attemptId) }
+
+ memoryUsage match {
+ case Some(m) =>
+ currentUsage.addAndGet(-m)
+ logInfo(s"Released ${Utils.bytesToString(m)} memory usage for " +
+ s"app $appId / $attemptId")
+ case None =>
+ }
+ }
+
+ private def approximateMemoryUsage(eventLogSize: Long, isCompressed: Boolean): Long = {
+ if (isCompressed) {
+ eventLogSize * 2
Review comment:
This memory usage approximation follows the way we estimate disk usage in HistoryServerDiskManager. In disk manager, we estimate the disk usage based on the log file size. I did some experiments to compare the memory usage of hybridStore (measured through org.apache.spark.util.SizeEstimator) and the disk usage of levelDB on a set of log files. I found they are similar. So I think if disk usage approximation works, this kind of memory usage approximation should work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420462651
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,9 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_KVSTORE_ENABLED = ConfigBuilder("spark.history.store.hybridKVStore.enabled")
+ .version("3.0.1")
Review comment:
Will address it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654478253
**[Test build #125136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125136/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-636834685
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654614124
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125163/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656689382
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656689382
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644482579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-624428013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r438130274
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
Review comment:
Ah ok i missed that thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658208340
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654837665
**[Test build #125219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125219/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r451705062
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
Drive by comment - given I added something similar to an in-house patch.
Add a `write(Iterator<E> values)` to kv store - this should make this switch order(s) faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-640924422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r437780958
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * A implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the writing to InMemoryStore is completed.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private[history] val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ val store = getStore()
+ store.write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
Review comment:
May be worth mentioning CacheQuantile in the comment as it can be recalculated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641545654
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641951221
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656158596
Yeah I’m OK either way. I think both ways wouldn’t bring major issues in reality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656594980
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656811985
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-642055084
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-636909048
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654344387
thanks for getting the numbers. yeah in general I agree, I would rather overestimate the memory usage as GC and OOM are bad. in practice if people find it using way less memory for their use cases they could make spark.history.store.hybridStore.maxMemoryUsage larger then real to fit more apps.
One question I do have is how much is memory usage affected by like the number of tasks? I imagine that could be a lot of the usage here from 1000 tasks. I'm curious for instance if someone has the use case of just a lot of very small applications. I guess it could go the other way as well though, if cluster mainly has huge jobs where you have 50000 tasks per job
without more stats I would think 10x for Zstd and 4x for others seems pretty safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r452534098
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.util.Collection
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.util.kvstore._
+
+/**
+ * An implementation of KVStore that accelerates event logs loading.
+ *
+ * When rebuilding the application state from event logs, HybridStore will
+ * write data to InMemoryStore at first and use a background thread to dump
+ * data to LevelDB once the app store is restored. We don't expect write
+ * operations (except the case for caching) after calling switch to level DB.
+ */
+
+private[history] class HybridStore extends KVStore {
+
+ private val inMemoryStore = new InMemoryStore()
+
+ private var levelDB: LevelDB = null
+
+ // Flag to indicate whether we should use inMemoryStore or levelDB
+ private val shouldUseInMemoryStore = new AtomicBoolean(true)
+
+ // Flag to indicate whether this hybrid store is closed, use this flag
+ // to avoid starting background thread after the store is closed
+ private val closed = new AtomicBoolean(false)
+
+ // A background thread that dumps data from inMemoryStore to levelDB
+ private var backgroundThread: Thread = null
+
+ // A hash map that stores all classes that had been writen to inMemoryStore
+ private val klassMap = new ConcurrentHashMap[Class[_], Boolean]
+
+ override def getMetadata[T](klass: Class[T]): T = {
+ getStore().getMetadata(klass)
+ }
+
+ override def setMetadata(value: Object): Unit = {
+ getStore().setMetadata(value)
+ }
+
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ getStore().read(klass, naturalKey)
+ }
+
+ override def write(value: Object): Unit = {
+ getStore().write(value)
+
+ if (backgroundThread == null) {
+ // New classes won't be dumped once the background thread is started
+ klassMap.putIfAbsent(value.getClass(), true)
+ }
+ }
+
+ override def delete(klass: Class[_], naturalKey: Object): Unit = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("delete() shouldn't be called after " +
+ "the hybrid store begins switching to levelDB")
+ }
+
+ getStore().delete(klass, naturalKey)
+ }
+
+ override def view[T](klass: Class[T]): KVStoreView[T] = {
+ getStore().view(klass)
+ }
+
+ override def count(klass: Class[_]): Long = {
+ getStore().count(klass)
+ }
+
+ override def count(klass: Class[_], index: String, indexedValue: Object): Long = {
+ getStore().count(klass, index, indexedValue)
+ }
+
+ override def close(): Unit = {
+ try {
+ closed.set(true)
+ if (backgroundThread != null && backgroundThread.isAlive()) {
+ // The background thread is still running, wait for it to finish
+ backgroundThread.join()
+ }
+ } finally {
+ inMemoryStore.close()
+ if (levelDB != null) {
+ levelDB.close()
+ }
+ }
+ }
+
+ override def removeAllByIndexValues[T](
+ klass: Class[T],
+ index: String,
+ indexValues: Collection[_]): Boolean = {
+ if (backgroundThread != null) {
+ throw new IllegalStateException("removeAllByIndexValues() shouldn't be " +
+ "called after the hybrid store begins switching to levelDB")
+ }
+
+ getStore().removeAllByIndexValues(klass, index, indexValues)
+ }
+
+ def setLevelDB(levelDB: LevelDB): Unit = {
+ this.levelDB = levelDB
+ }
+
+ /**
+ * This method is called when the writing is done for inMemoryStore. A
+ * background thread will be created and be started to dump data in inMemoryStore
+ * to levelDB. Once the dumping is completed, the underlying kvstore will be
+ * switched to levelDB.
+ */
+ def switchToLevelDB(
+ listener: HybridStore.SwitchToLevelDBListener,
+ appId: String,
+ attemptId: Option[String]): Unit = {
+ if (closed.get) {
+ return
+ }
+
+ backgroundThread = new Thread(() => {
+ try {
+ for (klass <- klassMap.keys().asScala) {
+ val it = inMemoryStore.view(klass).closeableIterator()
+ while (it.hasNext()) {
+ levelDB.write(it.next())
+ }
Review comment:
Sounds good. We can improve that afterward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656594552
**[Test build #125598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125598/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644442574
**[Test build #124078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124078/testReport)** for PR 28412 at commit [`3412d29`](https://github.com/apache/spark/commit/3412d294a76a7348377f4d1a87327dc4fe510e96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-644440678
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654994212
**[Test build #125219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125219/testReport)** for PR 28412 at commit [`b71a923`](https://github.com/apache/spark/commit/b71a923e4554e814ee923b64d1b22ce42825378c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-656389133
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-641475152
**[Test build #123702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123702/testReport)** for PR 28412 at commit [`3494307`](https://github.com/apache/spark/commit/3494307edea454bf37008f4db1c23ee16263b871).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] redsanket commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
redsanket commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r436146548
##########
File path: core/src/main/scala/org/apache/spark/internal/config/History.scala
##########
@@ -195,4 +195,18 @@ private[spark] object History {
.version("3.0.0")
.booleanConf
.createWithDefault(true)
+
+ val HYBRID_STORE_ENABLED = ConfigBuilder("spark.history.store.hybridStore.enabled")
+ .doc("Whether to use HybridStore as the store when parsing event logs. " +
+ "HybridStore will first write data to an in-memory store and having a background thread " +
+ "that dumps data to a disk store after the writing to in-memory store is completed. " +
+ "Use it with caution, as in-memory store requires higher memory usage.")
Review comment:
Do we need to still use this with caution as we are managing memory now? Do you still see any corner cases with it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-658453366
Thanks a lot for your reviews!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] baohe-zhang commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on a change in pull request #28412:
URL: https://github.com/apache/spark/pull/28412#discussion_r420461909
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -1167,6 +1168,58 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
// At this point the disk data either does not exist or was deleted because it failed to
// load, so the event log needs to be replayed.
+ // TODO: Maybe need to do other check to see if there's enough memory to
+ // use inMemoryStore.
+ if (hybridKVStoreEnabled) {
+ logInfo("Using HybridKVStore as KVStore")
+ var retried = false
+ var store: HybridKVStore = null
+ while(store == null) {
+ val reader = EventLogFileReader(fs, new Path(logDir, attempt.logPath),
+ attempt.lastIndex)
+ val isCompressed = reader.compressionCodec.isDefined
+ logInfo(s"Leasing disk manager space for app $appId / ${attempt.info.attemptId}...")
+ val lease = dm.lease(reader.totalSize, isCompressed)
+ try {
+ val s = new HybridKVStore()
+ val levelDB = KVUtils.open(lease.tmpPath, metadata)
+ s.setLevelDB(levelDB)
+
+ s.startBackgroundThreadToWriteToDB(new HybridKVStore.SwitchingToLevelDBListener {
+ override def onSwitchingToLevelDBSuccess: Unit = {
+ levelDB.close()
+ val newStorePath = lease.commit(appId, attempt.info.attemptId)
+ s.setLevelDB(KVUtils.open(newStorePath, metadata))
+ logInfo(s"Completely switched to use leveldb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ }
+
+ override def onSwitchingToLevelDBFail(e: Exception): Unit = {
+ logWarning(s"Failed to switch to use LevelDb for app" +
+ s" $appId / ${attempt.info.attemptId}")
+ levelDB.close()
+ throw e
+ }
+ })
+
+ rebuildAppStore(s, reader, attempt.info.lastUpdated.getTime())
+ s.stopBackgroundThreadAndSwitchToLevelDB()
+ store = s
+ } catch {
+ case _: IOException if !retried =>
+ // compaction may touch the file(s) which app rebuild wants to read
+ // compaction wouldn't run in short interval, so try again...
+ logWarning(s"Exception occurred while rebuilding app $appId - trying again...")
+ lease.rollback()
Review comment:
I didn't quite understand this comment, could you elaborate more? I think in the current implementation, if any exceptions are thrown when migrating data to leveldb, the hybrid kvstore will not switch to leveldb and the getStore() method in hybrid kvstore will always return an in-memory kvstore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org