You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zhangyue19921010 (via GitHub)" <gi...@apache.org> on 2025/04/08 07:54:45 UTC
[PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
zhangyue19921010 opened a new pull request, #13106:
URL: https://github.com/apache/hudi/pull/13106
### Change Logs
Enhance PartitionBucketIndexManager Support
1. Show Config Commands
2. Rollback Bucket Rescale Command
3. Dry-Run
4. Add Expression Command
### Impact
PartitionBucketIndexManager
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785729249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44911bca9b6433fa573adeb441a1a117858d671f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709)
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785930003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13106:
URL: https://github.com/apache/hudi/pull/13106#discussion_r2038595048
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/PartitionBucketIndexManager.scala:
##########
@@ -179,76 +181,88 @@ class PartitionBucketIndexManager extends BaseProcedure
// get partitions need to be rescaled
val rescalePartitionsMap = getDifferentPartitions(partition2BucketWithNewHashingConfig.asScala, partition2BucketWithLatestHashingConfig.asScala)
- val partitionsToRescale = rescalePartitionsMap.keys
-
- // get all fileSlices need to read
- val allFilesMap = FSUtils.getFilesInPartitions(context, metaClient.getStorage(), HoodieMetadataConfig.newBuilder.enable(mdtEnable).build,
- metaClient.getBasePath.toString, partitionsToRescale.map(relative => {
- new StoragePath(basePath, relative)
- }).map(storagePath => storagePath.toString).toArray)
- val files = allFilesMap.values().asScala.flatMap(x => x.asScala).toList
- val view = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline, files.asJava)
- val allFileSlice = partitionsToRescale.flatMap(partitionPath => {
- view.getLatestFileSlices(partitionPath).iterator().asScala
- }).toList
-
- // read all fileSlice para and get DF
- var tableSchemaWithMetaFields: Schema = null
- try tableSchemaWithMetaFields = HoodieAvroUtils.addMetadataFields(new TableSchemaResolver(metaClient).getTableAvroSchema(false), false)
- catch {
- case e: Exception =>
- throw new HoodieException("Failed to get table schema during clustering", e)
- }
+ if (dryRun) {
+ logInfo("Dry run OVERWRITE")
+ val content = rescalePartitionsMap.map(entry => {
+ val details =
+ s"""
+ |${entry._1} => ${entry._2}
+ |""".stripMargin
+ details
+ }).toSeq.mkString(";")
+ Seq(Row("SUCCESS", "DRY_RUN_OVERWRITE", content))
+ } else {
+ logInfo("Perform OVERWRITE with dry-run disabled.")
+ val partitionsToRescale = rescalePartitionsMap.keys
+ // get all fileSlices need to read
+ val allFilesMap = FSUtils.getFilesInPartitions(context, metaClient.getStorage(), HoodieMetadataConfig.newBuilder.enable(mdtEnable).build,
+ metaClient.getBasePath.toString, partitionsToRescale.map(relative => {
+ new StoragePath(basePath, relative)
+ }).map(storagePath => storagePath.toString).toArray)
+ val files = allFilesMap.values().asScala.flatMap(x => x.asScala).toList
+ val view = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline, files.asJava)
+ val allFileSlice = partitionsToRescale.flatMap(partitionPath => {
+ view.getLatestFileSlices(partitionPath).iterator().asScala
+ }).toList
+
+ // read all fileSlice para and get DF
+ var tableSchemaWithMetaFields: Schema = null
+ try tableSchemaWithMetaFields = HoodieAvroUtils.addMetadataFields(new TableSchemaResolver(metaClient).getTableAvroSchema(false), false)
+ catch {
+ case e: Exception =>
+ throw new HoodieException("Failed to get table schema during clustering", e)
+ }
- // broadcast reader context.
- val broadcastManager = new SparkBroadcastManager(context, metaClient)
- broadcastManager.prepareAndBroadcast()
- val sparkSchemaWithMetaFields = AvroConversionUtils.convertAvroSchemaToStructType(tableSchemaWithMetaFields)
+ // broadcast reader context.
+ val broadcastManager = new SparkBroadcastManager(context, metaClient)
+ broadcastManager.prepareAndBroadcast()
+ val sparkSchemaWithMetaFields = AvroConversionUtils.convertAvroSchemaToStructType(tableSchemaWithMetaFields)
- val res: RDD[InternalRow] = if (allFileSlice.isEmpty) {
- spark.sparkContext.emptyRDD
- } else {
- val serializableTableSchemaWithMetaFields = new SerializableSchema(tableSchemaWithMetaFields)
- val latestInstantTime = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
-
- spark.sparkContext.parallelize(allFileSlice, allFileSlice.size).flatMap(fileSlice => {
- // instantiate other supporting cast
- val readerSchema = serializableTableSchemaWithMetaFields.get
- val readerContextOpt = broadcastManager.retrieveFileGroupReaderContext(basePath)
- val internalSchemaOption: Option[InternalSchema] = Option.empty()
- // instantiate FG reader
- val fileGroupReader = new HoodieFileGroupReader(readerContextOpt.get(),
- metaClient.getStorage,
- basePath.toString,
- latestInstantTime.requestedTime(),
- fileSlice,
- readerSchema,
- readerSchema,
- internalSchemaOption, // not support evolution of schema for now
- metaClient,
- metaClient.getTableConfig.getProps,
- 0,
- java.lang.Long.MAX_VALUE,
- HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.defaultValue(),
- false)
- fileGroupReader.initRecordIterators()
- val iterator = fileGroupReader.getClosableIterator.asInstanceOf[HoodieFileGroupReader.HoodieFileGroupReaderIterator[InternalRow]]
- iterator.asScala
- })
- }
- val dataFrame = HoodieUnsafeUtils.createDataFrameFromRDD(sparkSession, res, sparkSchemaWithMetaFields)
- logInfo("Start to do bucket rescale for " + rescalePartitionsMap)
- val (success, _, _, _, _, _) = HoodieSparkSqlWriter.write(
- sparkSession.sqlContext,
- SaveMode.Append,
- finalConfig,
- dataFrame)
+ val res: RDD[InternalRow] = if (allFileSlice.isEmpty) {
+ spark.sparkContext.emptyRDD
+ } else {
+ val serializableTableSchemaWithMetaFields = new SerializableSchema(tableSchemaWithMetaFields)
+ val latestInstantTime = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
+
+ spark.sparkContext.parallelize(allFileSlice, allFileSlice.size).flatMap(fileSlice => {
+ // instantiate other supporting cast
+ val readerSchema = serializableTableSchemaWithMetaFields.get
+ val readerContextOpt = broadcastManager.retrieveFileGroupReaderContext(basePath)
+ val internalSchemaOption: Option[InternalSchema] = Option.empty()
+ // instantiate FG reader
+ val fileGroupReader = new HoodieFileGroupReader(readerContextOpt.get(),
+ metaClient.getStorage,
+ basePath.toString,
+ latestInstantTime.requestedTime(),
+ fileSlice,
+ readerSchema,
+ readerSchema,
+ internalSchemaOption, // not support evolution of schema for now
+ metaClient,
+ metaClient.getTableConfig.getProps,
+ 0,
+ java.lang.Long.MAX_VALUE,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.defaultValue(),
+ false)
+ fileGroupReader.initRecordIterators()
+ val iterator = fileGroupReader.getClosableIterator.asInstanceOf[HoodieFileGroupReader.HoodieFileGroupReaderIterator[InternalRow]]
+ iterator.asScala
+ })
+ }
+ val dataFrame = HoodieUnsafeUtils.createDataFrameFromRDD(sparkSession, res, sparkSchemaWithMetaFields)
+ logInfo("Start to do bucket rescale for " + rescalePartitionsMap)
+ val (success, _, _, _, _, _) = HoodieSparkSqlWriter.write(
+ sparkSession.sqlContext,
+ SaveMode.Append,
+ finalConfig,
+ dataFrame)
- val details = s"Expression: $expression, Bucket Number: $bucketNumber, Dry Run: $dryRun"
+ val details = s"Expression: $expression, Bucket Number: $bucketNumber, Dry Run: $dryRun"
Review Comment:
is the option val case-sensitive from users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785933289
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2796222607
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714",
"triggerID" : "2785933289",
"triggerType" : "MANUAL"
}, {
"hash" : "325f5d48664d239459240be5761d80e589e1e077",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4805",
"triggerID" : "325f5d48664d239459240be5761d80e589e1e077",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714)
* 325f5d48664d239459240be5761d80e589e1e077 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4805)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785934417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714",
"triggerID" : "2785933289",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2796529960
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714",
"triggerID" : "2785933289",
"triggerType" : "MANUAL"
}, {
"hash" : "325f5d48664d239459240be5761d80e589e1e077",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4805",
"triggerID" : "325f5d48664d239459240be5761d80e589e1e077",
"triggerType" : "PUSH"
}, {
"hash" : "325f5d48664d239459240be5761d80e589e1e077",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4808",
"triggerID" : "2796529124",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 325f5d48664d239459240be5761d80e589e1e077 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4805) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4808)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2786210011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714",
"triggerID" : "2785933289",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785571843
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44911bca9b6433fa573adeb441a1a117858d671f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13106:
URL: https://github.com/apache/hudi/pull/13106#discussion_r2038589081
##########
hudi-common/src/main/java/org/apache/hudi/common/model/PartitionBucketIndexHashingConfig.java:
##########
@@ -221,20 +244,19 @@ private static Option<String> getHashingConfigInstantToLoadBeforeOrOn(List<Strin
}
public static PartitionBucketIndexHashingConfig loadingLatestHashingConfig(HoodieTableMetaClient metaClient) {
- Option<String> instantToLoad = getHashingConfigInstantToLoad(metaClient, Option.empty());
- ValidationUtils.checkArgument(instantToLoad.isPresent(), "Can not load latest hashing config " + instantToLoad);
- Option<PartitionBucketIndexHashingConfig> latestHashingConfig = loadHashingConfig(metaClient.getStorage(), getHashingConfigPath(metaClient.getBasePath().toString(), instantToLoad.get()));
- ValidationUtils.checkArgument(latestHashingConfig.isPresent(), "Can not load latest hashing config " + instantToLoad);
+ Option<StoragePath> hashingConfigToLoad = getHashingConfigToLoad(metaClient, Option.empty());
Review Comment:
hashingConfigPathToLoad
##########
hudi-common/src/main/java/org/apache/hudi/common/model/PartitionBucketIndexHashingConfig.java:
##########
@@ -221,20 +244,19 @@ private static Option<String> getHashingConfigInstantToLoadBeforeOrOn(List<Strin
}
public static PartitionBucketIndexHashingConfig loadingLatestHashingConfig(HoodieTableMetaClient metaClient) {
- Option<String> instantToLoad = getHashingConfigInstantToLoad(metaClient, Option.empty());
- ValidationUtils.checkArgument(instantToLoad.isPresent(), "Can not load latest hashing config " + instantToLoad);
- Option<PartitionBucketIndexHashingConfig> latestHashingConfig = loadHashingConfig(metaClient.getStorage(), getHashingConfigPath(metaClient.getBasePath().toString(), instantToLoad.get()));
- ValidationUtils.checkArgument(latestHashingConfig.isPresent(), "Can not load latest hashing config " + instantToLoad);
+ Option<StoragePath> hashingConfigToLoad = getHashingConfigToLoad(metaClient, Option.empty());
+ ValidationUtils.checkArgument(hashingConfigToLoad.isPresent(), "Can not load latest hashing config " + hashingConfigToLoad);
+ Option<PartitionBucketIndexHashingConfig> latestHashingConfig = loadHashingConfig(metaClient.getStorage(), hashingConfigToLoad.get());
+ ValidationUtils.checkArgument(latestHashingConfig.isPresent(), "Can not load latest hashing config " + hashingConfigToLoad);
return latestHashingConfig.get();
}
public static Option<PartitionBucketIndexHashingConfig> loadingLatestHashingConfigBeforeOrOn(HoodieTableMetaClient metaClient, String instant) {
- Option<String> hashingConfigInstantToLoad = getHashingConfigInstantToLoad(metaClient, Option.of(instant));
- if (hashingConfigInstantToLoad.isPresent()) {
- Option<PartitionBucketIndexHashingConfig> latestHashingConfig = loadHashingConfig(metaClient.getStorage(),
- getHashingConfigPath(metaClient.getBasePath().toString(), hashingConfigInstantToLoad.get()));
- ValidationUtils.checkArgument(latestHashingConfig.isPresent(), "Can not load hashing config " + hashingConfigInstantToLoad + " based on " + instant);
+ Option<StoragePath> hashingConfigToLoad = getHashingConfigToLoad(metaClient, Option.of(instant));
Review Comment:
hashingConfigPathToLoad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2796529124
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785567859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44911bca9b6433fa573adeb441a1a117858d671f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13106:
URL: https://github.com/apache/hudi/pull/13106#discussion_r2039117964
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/PartitionBucketIndexManager.scala:
##########
@@ -179,76 +181,88 @@ class PartitionBucketIndexManager extends BaseProcedure
// get partitions need to be rescaled
val rescalePartitionsMap = getDifferentPartitions(partition2BucketWithNewHashingConfig.asScala, partition2BucketWithLatestHashingConfig.asScala)
- val partitionsToRescale = rescalePartitionsMap.keys
-
- // get all fileSlices need to read
- val allFilesMap = FSUtils.getFilesInPartitions(context, metaClient.getStorage(), HoodieMetadataConfig.newBuilder.enable(mdtEnable).build,
- metaClient.getBasePath.toString, partitionsToRescale.map(relative => {
- new StoragePath(basePath, relative)
- }).map(storagePath => storagePath.toString).toArray)
- val files = allFilesMap.values().asScala.flatMap(x => x.asScala).toList
- val view = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline, files.asJava)
- val allFileSlice = partitionsToRescale.flatMap(partitionPath => {
- view.getLatestFileSlices(partitionPath).iterator().asScala
- }).toList
-
- // read all fileSlice para and get DF
- var tableSchemaWithMetaFields: Schema = null
- try tableSchemaWithMetaFields = HoodieAvroUtils.addMetadataFields(new TableSchemaResolver(metaClient).getTableAvroSchema(false), false)
- catch {
- case e: Exception =>
- throw new HoodieException("Failed to get table schema during clustering", e)
- }
+ if (dryRun) {
+ logInfo("Dry run OVERWRITE")
+ val content = rescalePartitionsMap.map(entry => {
+ val details =
+ s"""
+ |${entry._1} => ${entry._2}
+ |""".stripMargin
+ details
+ }).toSeq.mkString(";")
+ Seq(Row("SUCCESS", "DRY_RUN_OVERWRITE", content))
+ } else {
+ logInfo("Perform OVERWRITE with dry-run disabled.")
+ val partitionsToRescale = rescalePartitionsMap.keys
+ // get all fileSlices need to read
+ val allFilesMap = FSUtils.getFilesInPartitions(context, metaClient.getStorage(), HoodieMetadataConfig.newBuilder.enable(mdtEnable).build,
+ metaClient.getBasePath.toString, partitionsToRescale.map(relative => {
+ new StoragePath(basePath, relative)
+ }).map(storagePath => storagePath.toString).toArray)
+ val files = allFilesMap.values().asScala.flatMap(x => x.asScala).toList
+ val view = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline, files.asJava)
+ val allFileSlice = partitionsToRescale.flatMap(partitionPath => {
+ view.getLatestFileSlices(partitionPath).iterator().asScala
+ }).toList
+
+ // read all fileSlice para and get DF
+ var tableSchemaWithMetaFields: Schema = null
+ try tableSchemaWithMetaFields = HoodieAvroUtils.addMetadataFields(new TableSchemaResolver(metaClient).getTableAvroSchema(false), false)
+ catch {
+ case e: Exception =>
+ throw new HoodieException("Failed to get table schema during clustering", e)
+ }
- // broadcast reader context.
- val broadcastManager = new SparkBroadcastManager(context, metaClient)
- broadcastManager.prepareAndBroadcast()
- val sparkSchemaWithMetaFields = AvroConversionUtils.convertAvroSchemaToStructType(tableSchemaWithMetaFields)
+ // broadcast reader context.
+ val broadcastManager = new SparkBroadcastManager(context, metaClient)
+ broadcastManager.prepareAndBroadcast()
+ val sparkSchemaWithMetaFields = AvroConversionUtils.convertAvroSchemaToStructType(tableSchemaWithMetaFields)
- val res: RDD[InternalRow] = if (allFileSlice.isEmpty) {
- spark.sparkContext.emptyRDD
- } else {
- val serializableTableSchemaWithMetaFields = new SerializableSchema(tableSchemaWithMetaFields)
- val latestInstantTime = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
-
- spark.sparkContext.parallelize(allFileSlice, allFileSlice.size).flatMap(fileSlice => {
- // instantiate other supporting cast
- val readerSchema = serializableTableSchemaWithMetaFields.get
- val readerContextOpt = broadcastManager.retrieveFileGroupReaderContext(basePath)
- val internalSchemaOption: Option[InternalSchema] = Option.empty()
- // instantiate FG reader
- val fileGroupReader = new HoodieFileGroupReader(readerContextOpt.get(),
- metaClient.getStorage,
- basePath.toString,
- latestInstantTime.requestedTime(),
- fileSlice,
- readerSchema,
- readerSchema,
- internalSchemaOption, // not support evolution of schema for now
- metaClient,
- metaClient.getTableConfig.getProps,
- 0,
- java.lang.Long.MAX_VALUE,
- HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.defaultValue(),
- false)
- fileGroupReader.initRecordIterators()
- val iterator = fileGroupReader.getClosableIterator.asInstanceOf[HoodieFileGroupReader.HoodieFileGroupReaderIterator[InternalRow]]
- iterator.asScala
- })
- }
- val dataFrame = HoodieUnsafeUtils.createDataFrameFromRDD(sparkSession, res, sparkSchemaWithMetaFields)
- logInfo("Start to do bucket rescale for " + rescalePartitionsMap)
- val (success, _, _, _, _, _) = HoodieSparkSqlWriter.write(
- sparkSession.sqlContext,
- SaveMode.Append,
- finalConfig,
- dataFrame)
+ val res: RDD[InternalRow] = if (allFileSlice.isEmpty) {
+ spark.sparkContext.emptyRDD
+ } else {
+ val serializableTableSchemaWithMetaFields = new SerializableSchema(tableSchemaWithMetaFields)
+ val latestInstantTime = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
+
+ spark.sparkContext.parallelize(allFileSlice, allFileSlice.size).flatMap(fileSlice => {
+ // instantiate other supporting cast
+ val readerSchema = serializableTableSchemaWithMetaFields.get
+ val readerContextOpt = broadcastManager.retrieveFileGroupReaderContext(basePath)
+ val internalSchemaOption: Option[InternalSchema] = Option.empty()
+ // instantiate FG reader
+ val fileGroupReader = new HoodieFileGroupReader(readerContextOpt.get(),
+ metaClient.getStorage,
+ basePath.toString,
+ latestInstantTime.requestedTime(),
+ fileSlice,
+ readerSchema,
+ readerSchema,
+ internalSchemaOption, // not support evolution of schema for now
+ metaClient,
+ metaClient.getTableConfig.getProps,
+ 0,
+ java.lang.Long.MAX_VALUE,
+ HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS.defaultValue(),
+ false)
+ fileGroupReader.initRecordIterators()
+ val iterator = fileGroupReader.getClosableIterator.asInstanceOf[HoodieFileGroupReader.HoodieFileGroupReaderIterator[InternalRow]]
+ iterator.asScala
+ })
+ }
+ val dataFrame = HoodieUnsafeUtils.createDataFrameFromRDD(sparkSession, res, sparkSchemaWithMetaFields)
+ logInfo("Start to do bucket rescale for " + rescalePartitionsMap)
+ val (success, _, _, _, _, _) = HoodieSparkSqlWriter.write(
+ sparkSession.sqlContext,
+ SaveMode.Append,
+ finalConfig,
+ dataFrame)
- val details = s"Expression: $expression, Bucket Number: $bucketNumber, Dry Run: $dryRun"
+ val details = s"Expression: $expression, Bucket Number: $bucketNumber, Dry Run: $dryRun"
Review Comment:
As a config name, it is case-sensitive.
To make it more user-friendly, change the config name from `dryRun` to `dry_run` like other Hoodie call command configs such as `RollbackToSavepointProcedure => instant_time config name`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13106:
URL: https://github.com/apache/hudi/pull/13106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785692672
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44911bca9b6433fa573adeb441a1a117858d671f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709)
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2796687774
The Azure CI failure is a known env issue: https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=4805&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2796395092
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714",
"triggerID" : "2785933289",
"triggerType" : "MANUAL"
}, {
"hash" : "325f5d48664d239459240be5761d80e589e1e077",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4805",
"triggerID" : "325f5d48664d239459240be5761d80e589e1e077",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 325f5d48664d239459240be5761d80e589e1e077 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4805)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2785697442
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44911bca9b6433fa573adeb441a1a117858d671f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709)
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13106:
URL: https://github.com/apache/hudi/pull/13106#discussion_r2038586739
##########
hudi-common/src/main/java/org/apache/hudi/common/model/PartitionBucketIndexHashingConfig.java:
##########
@@ -198,21 +198,44 @@ public static Option<PartitionBucketIndexHashingConfig> loadHashingConfig(Hoodie
* Get Latest committed hashing config instant to load.
* If instant is empty, then return latest hashing config instant
*/
- public static Option<String> getHashingConfigInstantToLoad(HoodieTableMetaClient metaClient, Option<String> instant) {
+ public static Option<StoragePath> getHashingConfigToLoad(HoodieTableMetaClient metaClient, Option<String> instant) {
try {
+ String basePath = metaClient.getBasePath().toString();
List<String> allCommittedHashingConfig = getCommittedHashingConfigInstants(metaClient);
if (instant.isPresent()) {
- Option<String> res = getHashingConfigInstantToLoadBeforeOrOn(allCommittedHashingConfig, instant.get());
+ Option<StoragePath> res = getHashingConfigInstantToLoadBeforeOrOn(allCommittedHashingConfig, instant.get()).map(i -> {
+ return getHashingConfigPath(basePath, i);
+ });
// fall back to look up archived hashing config instant before return empty
- return res.isPresent() ? res : getHashingConfigInstantToLoadBeforeOrOn(getArchiveHashingConfigInstants(metaClient), instant.get());
+ return res.isPresent() ? res : getHashingConfigInstantToLoadBeforeOrOn(getArchiveHashingConfigInstants(metaClient), instant.get()).map(i -> {
+ return getArchiveHashingConfigPath(basePath, i);
+ });
} else {
- return Option.of(allCommittedHashingConfig.get(allCommittedHashingConfig.size() - 1));
+ return Option.of(allCommittedHashingConfig.get(allCommittedHashingConfig.size() - 1)).map(i -> {
+ return getHashingConfigPath(basePath, i);
+ });
}
} catch (Exception e) {
throw new HoodieException("Failed to get hashing config instant to load.", e);
}
}
+ public static List<PartitionBucketIndexHashingConfig> getAllHashingConfig(HoodieTableMetaClient metaClient) throws IOException {
+ String basePath = metaClient.getBasePath().toString();
+ List<StoragePath> allHashingConfig = getCommittedHashingConfigInstants(metaClient).stream().map(instant -> {
Review Comment:
allHashingConfig -> allHashingConfigPaths
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8990] PartitionBucketIndexManager Enhance [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13106:
URL: https://github.com/apache/hudi/pull/13106#issuecomment-2796217389
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44911bca9b6433fa573adeb441a1a117858d671f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4709",
"triggerID" : "44911bca9b6433fa573adeb441a1a117858d671f",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711",
"triggerID" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"triggerType" : "PUSH"
}, {
"hash" : "5cead399e9e02dbb33e9df878c7f990e0fe1be48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714",
"triggerID" : "2785933289",
"triggerType" : "MANUAL"
}, {
"hash" : "325f5d48664d239459240be5761d80e589e1e077",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "325f5d48664d239459240be5761d80e589e1e077",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cead399e9e02dbb33e9df878c7f990e0fe1be48 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4711) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4714)
* 325f5d48664d239459240be5761d80e589e1e077 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org