You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "li36909 (via GitHub)" <gi...@apache.org> on 2023/02/15 12:54:04 UTC
[GitHub] [hudi] li36909 opened a new pull request, #7964: [HUDI-5805] hive query on mor get empty result before compaction
li36909 opened a new pull request, #7964:
URL: https://github.com/apache/hudi/pull/7964
Change Logs
when a mor table write data with flink cdc only, then before compaction the partition will only have log file, and no base file. then befor compaction, hive query result will always be empty.
it's because when hive getSplit on a native table, hive will ignore a partition which only has files start with '.', and because hudi has not set storageHandle when sync hive meta, then hive treat it as native table.
Impact
make storageHandle as DefaultStorageHandler when sync hive meta
Risk level (write none, low medium or high below)
none
Documentation Update
none
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477177508
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208",
"triggerID" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "9699c8bef218a6dc64977acfac4a8929dfbc6a9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9699c8bef218a6dc64977acfac4a8929dfbc6a9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aed8cffab1f915790180de9b49188b0077e0e6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208)
* 9699c8bef218a6dc64977acfac4a8929dfbc6a9c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477204527
@li36909 : Can you look at failing PR validation when you get a chance ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477181871
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208",
"triggerID" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "9699c8bef218a6dc64977acfac4a8929dfbc6a9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15820",
"triggerID" : "9699c8bef218a6dc64977acfac4a8929dfbc6a9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aed8cffab1f915790180de9b49188b0077e0e6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208)
* 9699c8bef218a6dc64977acfac4a8929dfbc6a9c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15820)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] li36909 closed pull request #7964: [HUDI-5805] Hive query on mor get empty result before compaction
Posted by "li36909 (via GitHub)" <gi...@apache.org>.
li36909 closed pull request #7964: [HUDI-5805] Hive query on mor get empty result before compaction
URL: https://github.com/apache/hudi/pull/7964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1439723568
@li36909 : For a hudi table defined in HMS with Storage by clause "org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler"), will HudiInputFormat be honored when running Hive queries ?
We need to test with other query engines using HMS to ensure there is no breakage. Can you confirm by testing new table with storage by clause against Spark, Presto, Trino queries using HMS would be unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7964: [HUDI-5805] Hive query on mor get empty result before compaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477828242
Thanks for the double check, it's fine and could be a handy fix for user in Hive side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1475699802
Spark should not be affected I think, we just need to make sure the query through Hive returns the correct result, i.e. the decuplication works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7964: [HUDI-5805] Hive query on mor get empty result before compaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477251803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208",
"triggerID" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "9699c8bef218a6dc64977acfac4a8929dfbc6a9c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15820",
"triggerID" : "9699c8bef218a6dc64977acfac4a8929dfbc6a9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9699c8bef218a6dc64977acfac4a8929dfbc6a9c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15820)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1475930407
> @li36909 : For a hudi table defined in HMS with Storage by clause "org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler"), will HudiInputFormat be honored when running Hive queries ?
>
> We need to test with other query engines using HMS to ensure there is no breakage. Can you confirm by testing new table with storage by clause against Spark, Presto, Trino queries using HMS would be unaffected.
@bvaradar @li36909
test with presto hudi connector/presto hive connector/spark/hive all test passed!
@li36909 pls rebase this code thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] li36909 commented on pull request #7964: [HUDI-5805] Hive query on mor get empty result before compaction
Posted by "li36909 (via GitHub)" <gi...@apache.org>.
li36909 commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477358120
> @li36909 : Can you look at failing PR validation when you get a chance ?
> @li36909 There are test failures: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=15820&view=logs&j=dcedfe73-9485-5cc5-817a-73b61fc5dcb0&t=746585d8-b50a-55c3-26c5-517d93af9934&l=16611, please fix it.
ok, will fix later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1431391906
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208",
"triggerID" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aed8cffab1f915790180de9b49188b0077e0e6a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #7964:
URL: https://github.com/apache/hudi/pull/7964#discussion_r1142266503
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -140,6 +142,10 @@ public void createTable(String tableName, MessageType storageSchema, String inpu
newTb.setTableType(TableType.EXTERNAL_TABLE.toString());
}
+ if (!tableProperties.containsKey(hive_metastoreConstants.META_TABLE_STORAGE)) {
+ tableProperties.put(hive_metastoreConstants.META_TABLE_STORAGE, DefaultStorageHandler.class.getName());
+ }
+
Review Comment:
The row format and storage format are both working for tables that does not define any inputformat, Hudi table ways has a inputformat definition, so we are good here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] li36909 commented on a diff in pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "li36909 (via GitHub)" <gi...@apache.org>.
li36909 commented on code in PR #7964:
URL: https://github.com/apache/hudi/pull/7964#discussion_r1142825680
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -140,6 +142,10 @@ public void createTable(String tableName, MessageType storageSchema, String inpu
newTb.setTableType(TableType.EXTERNAL_TABLE.toString());
}
+ if (!tableProperties.containsKey(hive_metastoreConstants.META_TABLE_STORAGE)) {
+ tableProperties.put(hive_metastoreConstants.META_TABLE_STORAGE, DefaultStorageHandler.class.getName());
+ }
+
Review Comment:
thanks for the check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7964: [HUDI-5805] Hive query on mor get empty result before compaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477244089
@li36909 There are test failures: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=15820&view=logs&j=dcedfe73-9485-5cc5-817a-73b61fc5dcb0&t=746585d8-b50a-55c3-26c5-517d93af9934&l=16611, please fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1431380448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aed8cffab1f915790180de9b49188b0077e0e6a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1432068330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208",
"triggerID" : "6aed8cffab1f915790180de9b49188b0077e0e6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aed8cffab1f915790180de9b49188b0077e0e6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #7964:
URL: https://github.com/apache/hudi/pull/7964#discussion_r1141918105
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -140,6 +142,10 @@ public void createTable(String tableName, MessageType storageSchema, String inpu
newTb.setTableType(TableType.EXTERNAL_TABLE.toString());
}
+ if (!tableProperties.containsKey(hive_metastoreConstants.META_TABLE_STORAGE)) {
+ tableProperties.put(hive_metastoreConstants.META_TABLE_STORAGE, DefaultStorageHandler.class.getName());
+ }
+
Review Comment:
The related document from Hive about the storage handlers: https://cwiki.apache.org/confluence/display/Hive/StorageHandlers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] li36909 commented on pull request #7964: [HUDI-5805] Hive query on mor get empty result before compaction
Posted by "li36909 (via GitHub)" <gi...@apache.org>.
li36909 commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477777831
sorry, this pr conflict with hive, when syncPartition we face the below error, so this pr need to be close
Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: ALTER TABLE cannot be used for a non-native table test1_ro
at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.validateAlterTableType(DDLSemanticAnalyzer.java:1354)
at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTableAddParts(DDLSemanticAnalyzer.java:2847)
at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:285)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #7964:
URL: https://github.com/apache/hudi/pull/7964#discussion_r1141979465
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -140,6 +142,10 @@ public void createTable(String tableName, MessageType storageSchema, String inpu
newTb.setTableType(TableType.EXTERNAL_TABLE.toString());
}
+ if (!tableProperties.containsKey(hive_metastoreConstants.META_TABLE_STORAGE)) {
+ tableProperties.put(hive_metastoreConstants.META_TABLE_STORAGE, DefaultStorageHandler.class.getName());
+ }
+
Review Comment:
It seems that the storage handler options conflict with row format and storage format options, does Hudi need those options though, need to confirm the affects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] li36909 commented on pull request #7964: [HUDI-5805] hive query on mor get empty result before compaction
Posted by "li36909 (via GitHub)" <gi...@apache.org>.
li36909 commented on PR #7964:
URL: https://github.com/apache/hudi/pull/7964#issuecomment-1477174596
> > @li36909 : For a hudi table defined in HMS with Storage by clause "org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler"), will HudiInputFormat be honored when running Hive queries ?
> > We need to test with other query engines using HMS to ensure there is no breakage. Can you confirm by testing new table with storage by clause against Spark, Presto, Trino queries using HMS would be unaffected.
>
> @bvaradar @li36909 @danny0405 test with presto hudi connector/presto hive connector/spark/hive all test passed!
>
> @li36909 pls rebase this code thanks
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org