You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ericm-db (via GitHub)" <gi...@apache.org> on 2024/08/13 02:16:54 UTC
[PR] [WIP] TWS VCF Integration [spark]
ericm-db opened a new pull request, #47730:
URL: https://github.com/apache/spark/pull/47730
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718822944
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##########
@@ -63,11 +63,13 @@ private[sql] class RocksDBStateStoreProvider
keyStateEncoderSpec: KeyStateEncoderSpec,
useMultipleValuesPerKey: Boolean = false,
isInternal: Boolean = false): Unit = {
- val newColFamilyId = ColumnFamilyUtils.createColFamilyIfAbsent(colFamilyName, isInternal)
+ val colFamilyId = ColumnFamilyUtils.createColFamilyIfAbsent(colFamilyName, isInternal)
Review Comment:
Maybe just return a `Short` here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718964868
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreSuite.scala:
##########
@@ -1090,6 +1089,42 @@ class RocksDBStateStoreSuite extends StateStoreSuiteBase[RocksDBStateStoreProvid
}
}
+ test("verify that column family id is assigned correctly after removal") {
Review Comment:
Adding a test case in the TransformWithStateSuite might be tough at this point in time because we don't support removal/deleteIfExists and don't have access to the provider. We have existing test cases that do restart queries so we should be good as far as that goes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718010262
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##########
@@ -581,10 +592,11 @@ private[sql] class RocksDBStateStoreProvider
Option[Short] = {
verifyColFamilyCreationOrDeletion("create_col_family", colFamilyName, isInternal)
if (!checkColFamilyExists(colFamilyName)) {
- val newColumnFamilyId = colFamilyId.incrementAndGet().toShort
+ val newColumnFamilyId = (maxColumnFamilyId + 1).toShort
Review Comment:
Lets add a note saying that this id generation is deterministic across batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718796773
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -941,11 +950,73 @@ object RocksDBCheckpointMetadata {
}
}
- def apply(rocksDBFiles: Seq[RocksDBImmutableFile], numKeys: Long): RocksDBCheckpointMetadata = {
- val sstFiles = rocksDBFiles.collect { case file: RocksDBSstFile => file }
- val logFiles = rocksDBFiles.collect { case file: RocksDBLogFile => file }
+ // Apply method for cases without column family information
+ def apply(
Review Comment:
Without all these, we get some compiler errors about default constructor params
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718827858
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -482,12 +491,20 @@ class RocksDB(
}
/**
+ * @param columnFamilyMapping A map of column family names to their corresponding VCF IDs.
+ * @param maxColumnFamilyId The maximum VCF ID used for column families.
+ * @param colFamilyIdsChanged A flag indicating whether column family IDs have changed.
+ * If they have, we need to create a snapshot regardless of whether
+ * changelog checkpointing is enabled or not
* Commit all the updates made as a version to DFS. The steps it needs to do to commits are:
* - Flush all changes to disk
* - Create a RocksDB checkpoint in a new local dir
* - Sync the checkpoint dir files to DFS
*/
- def commit(): Long = {
+ def commit(
+ columnFamilyMapping: Map[String, Short] = Map.empty,
+ maxColumnFamilyId: Short = 0,
+ colFamilyIdsChanged: Boolean = false): Long = {
Review Comment:
lets rename as `forceSnapshot` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "jingz-db (via GitHub)" <gi...@apache.org>.
jingz-db commented on PR #47730:
URL: https://github.com/apache/spark/pull/47730#issuecomment-2291877544
Nits:
- We could probably change the PR title name a bit as we are now not loading col family Id from State schema for VCF, instead we do that in RocksDB metadata file.
- Could we enable this test in `TransformWithStateSuite` to see if it passes: https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateSuite.scala#L654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "jingz-db (via GitHub)" <gi...@apache.org>.
jingz-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718801474
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreSuite.scala:
##########
@@ -1026,15 +1026,14 @@ class RocksDBStateStoreSuite extends StateStoreSuiteBase[RocksDBStateStoreProvid
}
}
- // TODO SPARK-48796 after restart state id will not be the same
- ignore(s"get, put, iterator, commit, load with multiple column families") {
+ test(s"get, put, iterator, commit, load with multiple column families") {
Review Comment:
Nits:
- We could probably change the PR title name a bit as we are now not loading col family Id from State schema for VCF, instead we do that in RocksDB metadata file.
- Could we enable this test in `TransformWithStateSuite` to see if it passes: https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateSuite.scala#L654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718824305
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -482,12 +491,20 @@ class RocksDB(
}
/**
+ * @param columnFamilyMapping A map of column family names to their corresponding VCF IDs.
Review Comment:
Lets move the params below the comment ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718844167
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreSuite.scala:
##########
@@ -1090,6 +1089,42 @@ class RocksDBStateStoreSuite extends StateStoreSuiteBase[RocksDBStateStoreProvid
}
}
+ test("verify that column family id is assigned correctly after removal") {
Review Comment:
could we add a test to ensure that vcf ids are reused correctly on restart ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1717992420
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -233,7 +250,39 @@ class RocksDB(
changelogWriter.foreach(_.abort())
changelogWriter = Some(fileManager.getChangeLogWriter(version + 1, useColumnFamilies))
}
- this
+ (this, loadedMapping, maxColumnFamilyId)
+ }
+
+ private def getColumnFamilyMapping(): Map[String, Short] = {
+ colFamilyNameToIdMap.asScala.toMap
+ }
+
+ private def getMaxColumnFamilyId(): Short = {
+ maxColumnFamilyId.get().toShort
+ }
+
+ def updateColumnFamilyMapping(mapping: Option[Map[String, Short]]): Unit = {
+ mapping.foreach { map =>
+ // Check if map and colFamilyNameToIdMap are the same
+ // if so, we don't need to force a snapshot and put a map.
+ if (map.size == colFamilyNameToIdMap.size() &&
Review Comment:
this is not thread safe any more. We need to guard with a explicit mutex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718007090
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -72,11 +74,16 @@ class RocksDB(
loggingId: String = "",
useColumnFamilies: Boolean = false) extends Logging {
+ private val colFamilyNameToIdMap = new ConcurrentHashMap[String, Short]()
Review Comment:
Hmm - so we have this map in 2 places now ? in the store as well as the provider ? could we remove one of them ? maybe the one in the provider ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718008992
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -941,11 +950,73 @@ object RocksDBCheckpointMetadata {
}
}
- def apply(rocksDBFiles: Seq[RocksDBImmutableFile], numKeys: Long): RocksDBCheckpointMetadata = {
- val sstFiles = rocksDBFiles.collect { case file: RocksDBSstFile => file }
- val logFiles = rocksDBFiles.collect { case file: RocksDBLogFile => file }
+ // Apply method for cases without column family information
+ def apply(
Review Comment:
Do we need so many variations here for the apply method ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718833497
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -213,6 +219,9 @@ class RocksDB(
} else {
metadata.numKeys
}
+ loadedMapping = metadata.columnFamilyMapping
Review Comment:
Can we verify that this is b/w compatible ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718833757
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -233,7 +242,7 @@ class RocksDB(
changelogWriter.foreach(_.abort())
changelogWriter = Some(fileManager.getChangeLogWriter(version + 1, useColumnFamilies))
}
- this
+ (this, loadedMapping, maxColumnFamilyId)
Review Comment:
lets move the return value to a case class ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718820400
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -19,7 +19,8 @@ package org.apache.spark.sql.execution.streaming.state
import java.io.File
import java.util.Locale
-import java.util.concurrent.TimeUnit
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
Review Comment:
Could we remove this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db closed pull request #47730: [SPARK-48796][SS] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting
URL: https://github.com/apache/spark/pull/47730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48796][SS] Load Column Family Id from State Schema for VCF when restarting [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47730:
URL: https://github.com/apache/spark/pull/47730#discussion_r1718008265
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -889,11 +892,17 @@ object RocksDBFileManagerMetrics {
case class RocksDBCheckpointMetadata(
sstFiles: Seq[RocksDBSstFile],
logFiles: Seq[RocksDBLogFile],
- numKeys: Long) {
+ numKeys: Long,
+ columnFamilyMapping: Option[Map[String, Short]] = None,
+ maxColumnFamilyId: Option[Short] = None) {
+
+ require(columnFamilyMapping.isDefined == maxColumnFamilyId.isDefined,
+ "columnFamilyMapping and maxColumnFamilyId must either both be defined or both be None")
+
import RocksDBCheckpointMetadata._
def json: String = {
- // We turn this field into a null to avoid write a empty logFiles field in the json.
+ // We turn this field into a null to avoid write an empty logFiles field in the json.
Review Comment:
nit: `writing an empty`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org