You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/20 14:17:25 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
HeartSaVioR opened a new pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649
### What changes were proposed in this pull request?
This patch addresses the case where compact metadata file is read twice in FileStreamSource during restarting query.
When restarting the query, there is a case which the query starts from compaction batch, and the batch has source metadata file to read. One case is that the previous query succeeded to read from inputs, but not finalized the batch for various reasons.
The patch finds the latest compaction batch when restoring from metadata log, and put entries for the batch into the file entry cache which would avoid reading compact batch file twice.
FileStreamSourceLog doesn't know about offset / commit metadata in checkpoint so doesn't know which exactly batch to start from, but in practice, only couple of latest batches are candidates to
be started from when restarting query. We leverage the fact to skip calculation if possible.
### Why are the changes needed?
Spark incurs unnecessary cost on reading the compact metadata file twice on some case, which may not be ignorable when the query has been processed huge number of files so far.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
New UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613136567
**[Test build #121230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121230/testReport)** for PR 27649 at commit [`6406e36`](https://github.com/apache/spark/commit/6406e36eb34377983aaf113495ca16b1553317a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604897973
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25185/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605039941
**[Test build #120480 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120480/testReport)** for PR 27649 at commit [`6406e36`](https://github.com/apache/spark/commit/6406e36eb34377983aaf113495ca16b1553317a3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604402901
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120409/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r399122439
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ assert(fileEntryCache.containsKey(4L))
+
+ val metadata2 = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache2 = metadata2 invokePrivate _fileEntryCache()
+
+ // allFiles() doesn't restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.allFiles() === allFiles)
+ assert(!fileEntryCache2.containsKey(4L))
+
+ // restore() will restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.restore() === allFiles)
+ assert(fileEntryCache2.containsKey(4L))
+
+ (5 to 5 + FileStreamSourceLog.PREV_NUM_BATCHES_TO_READ_IN_RESTORE).foreach { batchId =>
+ metadata2.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles2 = metadata2.allFiles()
Review comment:
As I've seen this val only used in one place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r398562612
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
##########
@@ -122,8 +123,35 @@ class FileStreamSourceLog(
}
batches
}
+
+ def restore(): Array[FileEntry] = {
+ val files = allFiles()
+
+ // When restarting the query, there is a case which the query starts from compaction batch,
+ // and the batch has source metadata file to read. One case is that the previous query
+ // succeeded to read from inputs, but not finalized the batch for various reasons.
+ // The below code finds the latest compaction batch, and put entries for the batch into the
+ // file entry cache which would avoid reading compact batch file twice.
+ // It doesn't know about offset / commit metadata in checkpoint so doesn't know which exactly
+ // batch to start from, but in practice, only couple of latest batches are candidates to
+ // be started. We leverage the fact to skip calculation if possible.
+ files.lastOption.foreach { lastEntry =>
+ val latestBatchId = lastEntry.batchId
+ val latestCompactedBatchId = getAllValidBatches(latestBatchId, compactInterval)(0)
+ if (latestCompactedBatchId > 0 &&
Review comment:
Seems like it's not working when one set `spark.sql.streaming.fileSource.log.compactInterval` to `1`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604853577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25173/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605039479
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589477272
cc. @tdas @zsxwing @gaborgsomogyi
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589074965
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605041083
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120480/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605039479
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613136937
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25918/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589250519
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118720/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604897968
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r399061092
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ assert(fileEntryCache.containsKey(4L))
+
+ val metadata2 = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache2 = metadata2 invokePrivate _fileEntryCache()
+
+ // allFiles() doesn't restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.allFiles() === allFiles)
+ assert(!fileEntryCache2.containsKey(4L))
+
+ // restore() will restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.restore() === allFiles)
+ assert(fileEntryCache2.containsKey(4L))
+
+ (5 to 5 + FileStreamSourceLog.PREV_NUM_BATCHES_TO_READ_IN_RESTORE).foreach { batchId =>
+ metadata2.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles2 = metadata2.allFiles()
Review comment:
Not sure I understand about "inline" here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604900632
**[Test build #120480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120480/testReport)** for PR 27649 at commit [`6406e36`](https://github.com/apache/spark/commit/6406e36eb34377983aaf113495ca16b1553317a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613136924
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589074965
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604402892
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r398541727
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ assert(fileEntryCache.containsKey(4L))
+
+ val metadata2 = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache2 = metadata2 invokePrivate _fileEntryCache()
+
+ // allFiles() doesn't restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.allFiles() === allFiles)
+ assert(!fileEntryCache2.containsKey(4L))
+
+ // restore() will restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.restore() === allFiles)
+ assert(fileEntryCache2.containsKey(4L))
Review comment:
Similar here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605041083
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120480/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r399062762
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
##########
@@ -122,8 +123,35 @@ class FileStreamSourceLog(
}
batches
}
+
+ def restore(): Array[FileEntry] = {
+ val files = allFiles()
+
+ // When restarting the query, there is a case which the query starts from compaction batch,
+ // and the batch has source metadata file to read. One case is that the previous query
+ // succeeded to read from inputs, but not finalized the batch for various reasons.
+ // The below code finds the latest compaction batch, and put entries for the batch into the
+ // file entry cache which would avoid reading compact batch file twice.
+ // It doesn't know about offset / commit metadata in checkpoint so doesn't know which exactly
+ // batch to start from, but in practice, only couple of latest batches are candidates to
+ // be started. We leverage the fact to skip calculation if possible.
+ files.lastOption.foreach { lastEntry =>
+ val latestBatchId = lastEntry.batchId
+ val latestCompactedBatchId = getAllValidBatches(latestBatchId, compactInterval)(0)
+ if (latestCompactedBatchId > 0 &&
+ (latestBatchId - latestCompactedBatchId) < PREV_NUM_BATCHES_TO_READ_IN_RESTORE) {
Review comment:
I thought I forgot to explain, but looks like I explained already:
```
// It doesn't know about offset / commit metadata in checkpoint so doesn't know which exactly
// batch to start from, but in practice, only couple of latest batches are candidates to
// be started. We leverage the fact to skip calculation if possible.
```
`only couple of latest batches` is the threshold - I heuristically took 2 here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589074979
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23471/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604402901
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120409/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r397858474
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ fileEntryCache.containsKey(4)
Review comment:
Don't we need some kind of assertion here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589250506
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604892986
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604897968
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604853571
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604853577
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25173/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604900632
**[Test build #120480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120480/testReport)** for PR 27649 at commit [`6406e36`](https://github.com/apache/spark/commit/6406e36eb34377983aaf113495ca16b1553317a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605041070
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605039487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120478/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604897516
**[Test build #120478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120478/testReport)** for PR 27649 at commit [`175ba90`](https://github.com/apache/spark/commit/175ba90669662e620282b68932f53421e95e3874).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604892994
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120466/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604272130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604852961
**[Test build #120466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120466/testReport)** for PR 27649 at commit [`175ba90`](https://github.com/apache/spark/commit/175ba90669662e620282b68932f53421e95e3874).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589074979
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23471/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604897973
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25185/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613135403
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613222587
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604853571
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604272130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r398540695
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ assert(fileEntryCache.containsKey(4L))
Review comment:
I think it would be good to test that only 4 added:
```
(0 to 3).foreach { batchId =>
assert(!fileEntryCache.containsKey(batchId))
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604852961
**[Test build #120466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120466/testReport)** for PR 27649 at commit [`175ba90`](https://github.com/apache/spark/commit/175ba90669662e620282b68932f53421e95e3874).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604402892
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605038184
**[Test build #120478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120478/testReport)** for PR 27649 at commit [`175ba90`](https://github.com/apache/spark/commit/175ba90669662e620282b68932f53421e95e3874).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613136924
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r398594049
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
##########
@@ -122,8 +123,35 @@ class FileStreamSourceLog(
}
batches
}
+
+ def restore(): Array[FileEntry] = {
+ val files = allFiles()
+
+ // When restarting the query, there is a case which the query starts from compaction batch,
+ // and the batch has source metadata file to read. One case is that the previous query
+ // succeeded to read from inputs, but not finalized the batch for various reasons.
+ // The below code finds the latest compaction batch, and put entries for the batch into the
+ // file entry cache which would avoid reading compact batch file twice.
+ // It doesn't know about offset / commit metadata in checkpoint so doesn't know which exactly
+ // batch to start from, but in practice, only couple of latest batches are candidates to
+ // be started. We leverage the fact to skip calculation if possible.
+ files.lastOption.foreach { lastEntry =>
+ val latestBatchId = lastEntry.batchId
+ val latestCompactedBatchId = getAllValidBatches(latestBatchId, compactInterval)(0)
+ if (latestCompactedBatchId > 0 &&
+ (latestBatchId - latestCompactedBatchId) < PREV_NUM_BATCHES_TO_READ_IN_RESTORE) {
Review comment:
Maybe a comment would be good why this heuristic is useful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r398576996
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ assert(fileEntryCache.containsKey(4L))
+
+ val metadata2 = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache2 = metadata2 invokePrivate _fileEntryCache()
+
+ // allFiles() doesn't restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.allFiles() === allFiles)
+ assert(!fileEntryCache2.containsKey(4L))
+
+ // restore() will restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.restore() === allFiles)
+ assert(fileEntryCache2.containsKey(4L))
+
+ (5 to 5 + FileStreamSourceLog.PREV_NUM_BATCHES_TO_READ_IN_RESTORE).foreach { batchId =>
+ metadata2.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles2 = metadata2.allFiles()
+
+ val metadata3 = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache3 = metadata3 invokePrivate _fileEntryCache()
+
+ // restore() will not restore the logs for the latest compact batch into file entry cache
+ // if the latest batch is too far from latest compact batch, because it's unlikely Spark
+ // will request the batch for the start point.
+ assert(metadata2.restore() === allFiles2)
+ assert(!fileEntryCache3.containsKey(4L))
Review comment:
Similar here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605039487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120478/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613222015
**[Test build #121230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121230/testReport)** for PR 27649 at commit [`6406e36`](https://github.com/apache/spark/commit/6406e36eb34377983aaf113495ca16b1553317a3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604895727
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604272137
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25117/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r398571490
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ assert(fileEntryCache.containsKey(4L))
+
+ val metadata2 = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache2 = metadata2 invokePrivate _fileEntryCache()
+
+ // allFiles() doesn't restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.allFiles() === allFiles)
+ assert(!fileEntryCache2.containsKey(4L))
+
+ // restore() will restore the logs for the latest compact batch into file entry cache
+ assert(metadata2.restore() === allFiles)
+ assert(fileEntryCache2.containsKey(4L))
+
+ (5 to 5 + FileStreamSourceLog.PREV_NUM_BATCHES_TO_READ_IN_RESTORE).foreach { batchId =>
+ metadata2.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles2 = metadata2.allFiles()
Review comment:
This can be inlined, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604892986
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589074348
**[Test build #118720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118720/testReport)** for PR 27649 at commit [`a5c4120`](https://github.com/apache/spark/commit/a5c41205a2262e14d0b8a284033b56debd24a90e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-605041070
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613222593
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121230/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604401360
**[Test build #120409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120409/testReport)** for PR 27649 at commit [`4e3046c`](https://github.com/apache/spark/commit/4e3046c2e3f94abf848f3cca53894e9ec3b61b4e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613222587
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604271665
**[Test build #120409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120409/testReport)** for PR 27649 at commit [`4e3046c`](https://github.com/apache/spark/commit/4e3046c2e3f94abf848f3cca53894e9ec3b61b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613222593
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121230/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r399060662
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
##########
@@ -122,8 +123,35 @@ class FileStreamSourceLog(
}
batches
}
+
+ def restore(): Array[FileEntry] = {
+ val files = allFiles()
+
+ // When restarting the query, there is a case which the query starts from compaction batch,
+ // and the batch has source metadata file to read. One case is that the previous query
+ // succeeded to read from inputs, but not finalized the batch for various reasons.
+ // The below code finds the latest compaction batch, and put entries for the batch into the
+ // file entry cache which would avoid reading compact batch file twice.
+ // It doesn't know about offset / commit metadata in checkpoint so doesn't know which exactly
+ // batch to start from, but in practice, only couple of latest batches are candidates to
+ // be started. We leverage the fact to skip calculation if possible.
+ files.lastOption.foreach { lastEntry =>
+ val latestBatchId = lastEntry.batchId
+ val latestCompactedBatchId = getAllValidBatches(latestBatchId, compactInterval)(0)
+ if (latestCompactedBatchId > 0 &&
Review comment:
That's just to prune the case where it may not help much, but yeah let's make it simple. It won't hurt in either way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589250519
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118720/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604897516
**[Test build #120478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120478/testReport)** for PR 27649 at commit [`175ba90`](https://github.com/apache/spark/commit/175ba90669662e620282b68932f53421e95e3874).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604892817
**[Test build #120466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120466/testReport)** for PR 27649 at commit [`175ba90`](https://github.com/apache/spark/commit/175ba90669662e620282b68932f53421e95e3874).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604271665
**[Test build #120409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120409/testReport)** for PR 27649 at commit [`4e3046c`](https://github.com/apache/spark/commit/4e3046c2e3f94abf848f3cca53894e9ec3b61b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613136937
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25918/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589249534
**[Test build #118720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118720/testReport)** for PR 27649 at commit [`a5c4120`](https://github.com/apache/spark/commit/a5c41205a2262e14d0b8a284033b56debd24a90e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r382027215
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
##########
@@ -122,8 +123,35 @@ class FileStreamSourceLog(
}
batches
}
+
+ def restore(): Array[FileEntry] = {
Review comment:
To not touch existing semantic of `allFiles()`, I simply add a new method to cover the new semantic. I'll just override `allFiles()` if it's preferred.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604850334
Thanks for reviewing. I guess I addressed review comments, please take a look again. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589381443
https://github.com/apache/spark/blob/fc4e56a54c15e20baf085e6061d3d83f5ce1185d/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala#L92-L95
We're describing reading from latest compacted log file as "quite time-consuming", and this patch addresses another case of reading from latest compacted log file, which would bring actual benefit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604272137
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25117/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604892994
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120466/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589250506
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-613136567
**[Test build #121230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121230/testReport)** for PR 27649 at commit [`6406e36`](https://github.com/apache/spark/commit/6406e36eb34377983aaf113495ca16b1553317a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-589074348
**[Test build #118720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118720/testReport)** for PR 27649 at commit [`a5c4120`](https://github.com/apache/spark/commit/a5c41205a2262e14d0b8a284033b56debd24a90e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #27649:
[SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log
twice if the query restarts from compact batch
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#discussion_r398358987
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1371,6 +1371,60 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("restore from file stream source log") {
+ def createEntries(batchId: Long, count: Int): Array[FileEntry] = {
+ (1 to count).map { idx =>
+ FileEntry(s"path_${batchId}_$idx", 10000 * batchId + count, batchId)
+ }.toArray
+ }
+
+ withSQLConf(SQLConf.FILE_SOURCE_LOG_COMPACT_INTERVAL.key -> "5") {
+ withTempDir { chk =>
+ val _fileEntryCache = PrivateMethod[java.util.LinkedHashMap[Long, Array[FileEntry]]](
+ Symbol("fileEntryCache"))
+
+ val metadata = new FileStreamSourceLog(FileStreamSourceLog.VERSION, spark,
+ chk.getCanonicalPath)
+ val fileEntryCache = metadata invokePrivate _fileEntryCache()
+
+ (0 to 4).foreach { batchId =>
+ metadata.add(batchId, createEntries(batchId, 100))
+ }
+ val allFiles = metadata.allFiles()
+
+ // batch 4 is a compact batch which logs would be cached in fileEntryCache
+ fileEntryCache.containsKey(4)
Review comment:
lol I have no idea why I missed assert here. Nice finding. Will add.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on issue #27649: [SPARK-30900][SS]
FileStreamSource: Avoid reading compact metadata log twice if the query
restarts from compact batch
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on issue #27649: [SPARK-30900][SS] FileStreamSource: Avoid reading compact metadata log twice if the query restarts from compact batch
URL: https://github.com/apache/spark/pull/27649#issuecomment-604895627
Seems unrelated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org