You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/11 00:46:21 UTC
[GitHub] [hudi] jonvex commented on a diff in pull request #7632: [HUDI-3775] Allow for offline compaction of MOR tables via spark streaming
jonvex commented on code in PR #7632:
URL: https://github.com/apache/hudi/pull/7632#discussion_r1066475735
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala:
##########
@@ -117,7 +119,8 @@ class HoodieStreamingSink(sqlContext: SQLContext,
retry(retryCnt, retryIntervalMs)(
Try(
HoodieSparkSqlWriter.write(
- sqlContext, mode, updatedOptions, data, hoodieTableConfig, writeClient, Some(triggerAsyncCompactor), Some(triggerAsyncClustering),
+ sqlContext, mode, updatedOptions, data, hoodieTableConfig, writeClient,
+ if (disableCompaction) None else Some(triggerAsyncCompactor), Some(triggerAsyncClustering),
Review Comment:
I attempted a test with the following code, but it doesn't work. If you know how to trigger the streaming that would be helpful. The destPath was never even created or populated with what I have below. I do have the commands that I used for testing this pr in the jira ticket so if I can't get this to work at least we have some documentation of testing.
```
@Test
def testStructuredStreamingWithDisabledCompaction(): Unit = {
val (sourcePath, destPath) = initStreamingSourceAndDestPath("source", "dest")
// First chunk of data
val records1 = recordsToStrings(dataGen.generateInserts("000", 100)).toList
val inputDF1 = spark.read.json(spark.sparkContext.parallelize(records1, 2))
val futures = new Array[Future[Unit]](25)
val opts = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.MERGE_ON_READ.name()) + (DataSourceWriteOptions.STREAMING_DISABLE_COMPACTION.key -> "true" )
futures(0) = initStreamingWriteFuture(inputDF1.schema, sourcePath, destPath, opts)
for (i <- 1 to 24) {
val records = recordsToStrings(dataGen.generateUpdates(String.format("%03d", new Integer(i)), 100)).toList
val inputDF = spark.read.json(spark.sparkContext.parallelize(records, 2))
futures(i) = initStreamingWriteFuture(inputDF.schema, sourcePath, destPath, opts)
}
waitTillAtleastNCommits(fs, destPath, 25, 1000, 5)
val metaClient = HoodieTableMetaClient.builder().setConf(fs.getConf).setBasePath(destPath)
.setLoadActiveTimelineOnLoad(true).build()
assertTrue(metaClient.getActiveTimeline.getCommitTimeline.empty())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org