You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2025/08/23 17:02:25 UTC
[PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
rahil-c opened a new pull request, #13756:
URL: https://github.com/apache/hudi/pull/13756
…degradation
### Change Logs
FGReader should take necessary information from the constructor/builder without having to do additional FS calls and scanning on the executor (e.g., loading table config, scanning timeline, reading schema, etc.) before reading the records.
Constructing meta client incurs loading table config, scanning timeline, reading schema, etc., which are very expensive operations, which must not happen per file group.
### Impact
Aim of PR is ensure that metaclient is not being created in each metaclient.
### Risk level (write none, low medium or high below)
low
### Documentation Update
No doc update
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #13756:
URL: https://github.com/apache/hudi/pull/13756#discussion_r2296853621
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,17 +211,19 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val engineContext = new HoodieSparkEngineContext(new JavaSparkContext(spark.sparkContext))
val maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(engineContext.getTaskContextSupplier, options.asJava)
+ // Create metaclient on driver to avoid expensive operations on executors
+ val storageConf = new HadoopStorageConfiguration(broadcastedStorageConf.value.value)
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient
+ .builder().setConf(storageConf).setBasePath(tablePath).build
Review Comment:
https://issues.apache.org/jira/browse/HUDI-9741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3217179209
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef6728737e5b16651a05ce2ec9f334cc7085459d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3221211860
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
}, {
"hash" : "6750195bc5df58f5129e1590560844a0cee475f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106",
"triggerID" : "6750195bc5df58f5129e1590560844a0cee475f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6750195bc5df58f5129e1590560844a0cee475f1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3218427131
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef6728737e5b16651a05ce2ec9f334cc7085459d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081)
* c98f8a093ad24ddaf3381e30f4a73d8e79be7e54 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3217237407
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef6728737e5b16651a05ce2ec9f334cc7085459d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #13756:
URL: https://github.com/apache/hudi/pull/13756#discussion_r2296386646
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,17 +211,19 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val engineContext = new HoodieSparkEngineContext(new JavaSparkContext(spark.sparkContext))
val maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(engineContext.getTaskContextSupplier, options.asJava)
+ // Create metaclient on driver to avoid expensive operations on executors
+ val storageConf = new HadoopStorageConfiguration(broadcastedStorageConf.value.value)
Review Comment:
let me try that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13756:
URL: https://github.com/apache/hudi/pull/13756#discussion_r2298595455
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -255,15 +254,15 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
case _ =>
readBaseFile(file, baseFileReader.value, requestedSchema, remainingPartitionSchema, fixedPartitionIndexes,
- requiredSchema, partitionSchema, outputSchema, filters, storageConf)
+ requiredSchema, partitionSchema, outputSchema, filters, augmentedStorageConf)
}
// CDC queries.
case hoodiePartitionCDCFileGroupSliceMapping: HoodiePartitionCDCFileGroupMapping =>
- buildCDCRecordIterator(hoodiePartitionCDCFileGroupSliceMapping, fileGroupBaseFileReader.value, storageConf, fileIndexProps, requiredSchema)
+ buildCDCRecordIterator(hoodiePartitionCDCFileGroupSliceMapping, fileGroupBaseFileReader.value, augmentedStorageConf, fileIndexProps, requiredSchema, metaClient)
Review Comment:
Similar here on using `storageConf`
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -255,15 +254,15 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
case _ =>
readBaseFile(file, baseFileReader.value, requestedSchema, remainingPartitionSchema, fixedPartitionIndexes,
- requiredSchema, partitionSchema, outputSchema, filters, storageConf)
+ requiredSchema, partitionSchema, outputSchema, filters, augmentedStorageConf)
}
// CDC queries.
case hoodiePartitionCDCFileGroupSliceMapping: HoodiePartitionCDCFileGroupMapping =>
- buildCDCRecordIterator(hoodiePartitionCDCFileGroupSliceMapping, fileGroupBaseFileReader.value, storageConf, fileIndexProps, requiredSchema)
+ buildCDCRecordIterator(hoodiePartitionCDCFileGroupSliceMapping, fileGroupBaseFileReader.value, augmentedStorageConf, fileIndexProps, requiredSchema, metaClient)
case _ =>
readBaseFile(file, baseFileReader.value, requestedSchema, remainingPartitionSchema, fixedPartitionIndexes,
- requiredSchema, partitionSchema, outputSchema, filters, storageConf)
+ requiredSchema, partitionSchema, outputSchema, filters, augmentedStorageConf)
Review Comment:
Similar
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -255,15 +254,15 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
case _ =>
readBaseFile(file, baseFileReader.value, requestedSchema, remainingPartitionSchema, fixedPartitionIndexes,
- requiredSchema, partitionSchema, outputSchema, filters, storageConf)
+ requiredSchema, partitionSchema, outputSchema, filters, augmentedStorageConf)
Review Comment:
This should still use `storageConf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3218428050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef6728737e5b16651a05ce2ec9f334cc7085459d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081)
* c98f8a093ad24ddaf3381e30f4a73d8e79be7e54 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3217180334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef6728737e5b16651a05ce2ec9f334cc7085459d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3221684830
@yihua can you merge this when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13756:
URL: https://github.com/apache/hudi/pull/13756#discussion_r2296276078
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,17 +211,19 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val engineContext = new HoodieSparkEngineContext(new JavaSparkContext(spark.sparkContext))
val maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(engineContext.getTaskContextSupplier, options.asJava)
+ // Create metaclient on driver to avoid expensive operations on executors
+ val storageConf = new HadoopStorageConfiguration(broadcastedStorageConf.value.value)
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient
+ .builder().setConf(storageConf).setBasePath(tablePath).build
Review Comment:
As a follow-up PR could we check if the caller can pass the meta client directly to `HoodieFileGroupReaderBasedFileFormat`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3221217324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
}, {
"hash" : "6750195bc5df58f5129e1590560844a0cee475f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106",
"triggerID" : "6750195bc5df58f5129e1590560844a0cee475f1",
"triggerType" : "PUSH"
}, {
"hash" : "a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6750195bc5df58f5129e1590560844a0cee475f1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106)
* a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3221394830
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
}, {
"hash" : "6750195bc5df58f5129e1590560844a0cee475f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106",
"triggerID" : "6750195bc5df58f5129e1590560844a0cee475f1",
"triggerType" : "PUSH"
}, {
"hash" : "a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8110",
"triggerID" : "a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6750195bc5df58f5129e1590560844a0cee475f1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106)
* a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8110)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3218524791
@yihua Let me know if you have any other concerns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3221206719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
}, {
"hash" : "6750195bc5df58f5129e1590560844a0cee475f1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106",
"triggerID" : "6750195bc5df58f5129e1590560844a0cee475f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c98f8a093ad24ddaf3381e30f4a73d8e79be7e54 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089)
* 6750195bc5df58f5129e1590560844a0cee475f1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3221201356
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
}, {
"hash" : "6750195bc5df58f5129e1590560844a0cee475f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6750195bc5df58f5129e1590560844a0cee475f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c98f8a093ad24ddaf3381e30f4a73d8e79be7e54 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089)
* 6750195bc5df58f5129e1590560844a0cee475f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3218466168
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c98f8a093ad24ddaf3381e30f4a73d8e79be7e54 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #13756:
URL: https://github.com/apache/hudi/pull/13756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13756:
URL: https://github.com/apache/hudi/pull/13756#discussion_r2296271457
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,17 +211,19 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val engineContext = new HoodieSparkEngineContext(new JavaSparkContext(spark.sparkContext))
val maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(engineContext.getTaskContextSupplier, options.asJava)
+ // Create metaclient on driver to avoid expensive operations on executors
+ val storageConf = new HadoopStorageConfiguration(broadcastedStorageConf.value.value)
Review Comment:
Use `augmentedStorageConf` since this happens on the driver and there is no need to do any broadcast?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3217178734
cc @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Avoid using meta client in FGReader to avoid performance … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #13756:
URL: https://github.com/apache/hudi/pull/13756#discussion_r2296839504
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,17 +211,19 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val engineContext = new HoodieSparkEngineContext(new JavaSparkContext(spark.sparkContext))
val maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(engineContext.getTaskContextSupplier, options.asJava)
+ // Create metaclient on driver to avoid expensive operations on executors
+ val storageConf = new HadoopStorageConfiguration(broadcastedStorageConf.value.value)
Review Comment:
@yihua Have pushed a commit to try using the `augmentedStorageConf`. I wonder thought if this is safe as I thought the reason of using this `val broadcastedStorageConf = spark.sparkContext.broadcast(new SerializableConfiguration(augmentedStorageConf.unwrap()))` was because of the fact that the `augmentedStorageConf` may not be serializable, due to its hadoop `Configuration` usage which does not implement .
If there is an issue with the recent commit then i think i might be good to switch back to what it was before.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,17 +211,19 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val engineContext = new HoodieSparkEngineContext(new JavaSparkContext(spark.sparkContext))
val maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(engineContext.getTaskContextSupplier, options.asJava)
+ // Create metaclient on driver to avoid expensive operations on executors
+ val storageConf = new HadoopStorageConfiguration(broadcastedStorageConf.value.value)
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient
+ .builder().setConf(storageConf).setBasePath(tablePath).build
Review Comment:
Let me make a follow up JIRA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9739] Create meta client in Spark driver for FGReader based format to avoid performance degradation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13756:
URL: https://github.com/apache/hudi/pull/13756#issuecomment-3221634640
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8081",
"triggerID" : "ef6728737e5b16651a05ce2ec9f334cc7085459d",
"triggerType" : "PUSH"
}, {
"hash" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8089",
"triggerID" : "c98f8a093ad24ddaf3381e30f4a73d8e79be7e54",
"triggerType" : "PUSH"
}, {
"hash" : "6750195bc5df58f5129e1590560844a0cee475f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8106",
"triggerID" : "6750195bc5df58f5129e1590560844a0cee475f1",
"triggerType" : "PUSH"
}, {
"hash" : "a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8110",
"triggerID" : "a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9d85cdf7e606a3c26a9a3d91150345e8c98b2d5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8110)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org