You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2024/05/25 08:12:57 UTC
[PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils i… [hudi]
yihua opened a new pull request, #11302:
URL: https://github.com/apache/hudi/pull/11302
…mplementation
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #11302:
URL: https://github.com/apache/hudi/pull/11302#issuecomment-2131831426
Azure CI is green.
<img width="1460" alt="Screenshot 2024-05-25 at 17 51 55" src="https://github.com/apache/hudi/assets/2497195/dafc4c2a-f367-4907-a718-2d2a7c19fcd3">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614727283
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkParquetReader.java:
##########
@@ -69,7 +69,7 @@ public HoodieSparkParquetReader(HoodieStorage storage, StoragePath path) {
this.storage = storage.newInstance(path, storage.getConf().newInstance());
// Avoid adding record in list element when convert parquet schema to avro schema
this.storage.getConf().set(ADD_LIST_ELEMENT_RECORDS, "false");
- this.parquetUtils = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET);
+ this.parquetUtils = new HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET);
Review Comment:
Yes. Given this Spark so I hardcoded it. Fix now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils i… [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614632735
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkParquetReader.java:
##########
@@ -69,7 +69,7 @@ public HoodieSparkParquetReader(HoodieStorage storage, StoragePath path) {
this.storage = storage.newInstance(path, storage.getConf().newInstance());
// Avoid adding record in list element when convert parquet schema to avro schema
this.storage.getConf().set(ADD_LIST_ELEMENT_RECORDS, "false");
- this.parquetUtils = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET);
+ this.parquetUtils = new HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET);
Review Comment:
can't we use `HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)`?
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -140,7 +141,8 @@ class SparkHelper(sqlContext: SQLContext, fs: FileSystem) {
* @return <pre>true</pre> if all keys are added to the bloom filter; <pre>false</pre> otherwise.
*/
def fileKeysAgainstBF(storage: HoodieStorage, sqlContext: SQLContext, file: String): Boolean = {
- val bf = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readBloomFilterFromMetadata(storage, new StoragePath(file))
+ val bf = new HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
same here
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapOperator.java:
##########
@@ -202,7 +203,8 @@ protected void loadRecords(String partitionPath) throws Exception {
Option<HoodieInstant> latestCommitTime = commitsTimeline.filterCompletedAndCompactionInstants().lastInstant();
if (latestCommitTime.isPresent()) {
- FileFormatUtils fileUtils = FileFormatUtils.getInstance(this.hoodieTable.getBaseFileFormat());
+ FileFormatUtils fileUtils = HoodieIOFactory.getIOFactory(hoodieTable.getStorage())
+ .getFileFormatUtils(this.hoodieTable.getBaseFileFormat());
Review Comment:
```suggestion
.getFileFormatUtils(hoodieTable.getBaseFileFormat());
```
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroParquetReader.java:
##########
@@ -68,7 +68,7 @@ public HoodieAvroParquetReader(HoodieStorage storage, StoragePath path) {
// by the Reader (for proper config propagation to Parquet components)
this.storage = storage.newInstance(path, tryOverrideDefaultConfigs(storage.getConf().newInstance()));
this.path = path;
- this.parquetUtils = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET);
+ this.parquetUtils = new HoodieHadoopIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET);
Review Comment:
same here, can we use `HoodieIOFactory` instead of specific impl of HoodieIOFactory?
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroOrcReader.java:
##########
@@ -58,7 +58,7 @@ public class HoodieAvroOrcReader extends HoodieAvroFileReader {
public HoodieAvroOrcReader(HoodieStorage storage, StoragePath path) {
this.storage = storage;
this.path = path;
- this.orcUtils = FileFormatUtils.getInstance(HoodieFileFormat.ORC);
+ this.orcUtils = new HoodieHadoopIOFactory(storage).getFileFormatUtils(HoodieFileFormat.ORC);
Review Comment:
same here, can we use `HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(HoodieFileFormat.ORC)`?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -1440,10 +1440,11 @@ public List<HoodieColumnRangeMetadata<Comparable>> getSortedColumnStatsList(
.collect(Collectors.toList());
} else {
return baseFileNameList.stream().flatMap(filename ->
- FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readColumnStatsFromMetadata(
- metaClient.getStorage(),
- new StoragePath(FSUtils.constructAbsolutePath(metaClient.getBasePathV2(), partitionPath), filename),
- allColumnNameList).stream())
+ new HoodieSparkIOFactory(metaClient.getStorage()).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
here also, let;s see if we can use `HoodieIOFactory`. And would be good to create the util before stream.
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -48,7 +48,8 @@ object SparkHelpers {
sourceFile: StoragePath,
destinationFile: StoragePath,
keysToSkip: Set[String]) {
- val sourceRecords = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readAvroRecords(storage, sourceFile).asScala
+ val sourceRecords = new HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
same comment, use `HoodieIOFactory` if posible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614729375
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -1440,10 +1440,11 @@ public List<HoodieColumnRangeMetadata<Comparable>> getSortedColumnStatsList(
.collect(Collectors.toList());
} else {
return baseFileNameList.stream().flatMap(filename ->
- FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readColumnStatsFromMetadata(
- metaClient.getStorage(),
- new StoragePath(FSUtils.constructAbsolutePath(metaClient.getBasePathV2(), partitionPath), filename),
- allColumnNameList).stream())
+ new HoodieSparkIOFactory(metaClient.getStorage()).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614728446
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -48,7 +48,8 @@ object SparkHelpers {
sourceFile: StoragePath,
destinationFile: StoragePath,
keysToSkip: Set[String]) {
- val sourceRecords = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readAvroRecords(storage, sourceFile).asScala
+ val sourceRecords = new HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614727596
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapOperator.java:
##########
@@ -202,7 +203,8 @@ protected void loadRecords(String partitionPath) throws Exception {
Option<HoodieInstant> latestCommitTime = commitsTimeline.filterCompletedAndCompactionInstants().lastInstant();
if (latestCommitTime.isPresent()) {
- FileFormatUtils fileUtils = FileFormatUtils.getInstance(this.hoodieTable.getBaseFileFormat());
+ FileFormatUtils fileUtils = HoodieIOFactory.getIOFactory(hoodieTable.getStorage())
+ .getFileFormatUtils(this.hoodieTable.getBaseFileFormat());
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614727946
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroOrcReader.java:
##########
@@ -58,7 +58,7 @@ public class HoodieAvroOrcReader extends HoodieAvroFileReader {
public HoodieAvroOrcReader(HoodieStorage storage, StoragePath path) {
this.storage = storage;
this.path = path;
- this.orcUtils = FileFormatUtils.getInstance(HoodieFileFormat.ORC);
+ this.orcUtils = new HoodieHadoopIOFactory(storage).getFileFormatUtils(HoodieFileFormat.ORC);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #11302:
URL: https://github.com/apache/hudi/pull/11302#issuecomment-2131311373
Thanks for addressing the comments. Please also update the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils i… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11302:
URL: https://github.com/apache/hudi/pull/11302#issuecomment-2131134599
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "049a4ffff479918567e33acfbd7417667858e500",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "049a4ffff479918567e33acfbd7417667858e500",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 049a4ffff479918567e33acfbd7417667858e500 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #11302:
URL: https://github.com/apache/hudi/pull/11302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614728639
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -140,7 +141,8 @@ class SparkHelper(sqlContext: SQLContext, fs: FileSystem) {
* @return <pre>true</pre> if all keys are added to the bloom filter; <pre>false</pre> otherwise.
*/
def fileKeysAgainstBF(storage: HoodieStorage, sqlContext: SQLContext, file: String): Boolean = {
- val bf = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET).readBloomFilterFromMetadata(storage, new StoragePath(file))
+ val bf = new HoodieSparkIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET)
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils implementation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11302:
URL: https://github.com/apache/hudi/pull/11302#discussion_r1614728159
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroParquetReader.java:
##########
@@ -68,7 +68,7 @@ public HoodieAvroParquetReader(HoodieStorage storage, StoragePath path) {
// by the Reader (for proper config propagation to Parquet components)
this.storage = storage.newInstance(path, tryOverrideDefaultConfigs(storage.getConf().newInstance()));
this.path = path;
- this.parquetUtils = FileFormatUtils.getInstance(HoodieFileFormat.PARQUET);
+ this.parquetUtils = new HoodieHadoopIOFactory(storage).getFileFormatUtils(HoodieFileFormat.PARQUET);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org