You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "KnightChess (via GitHub)" <gi...@apache.org> on 2024/08/06 05:05:36 UTC
[PR] [MINOR] make the read and write behavior consistent in spark [hudi]
KnightChess opened a new pull request, #11730:
URL: https://github.com/apache/hudi/pull/11730
### Change Logs
for spark, the `hoodie.metadata.enable` is enable when writer side, but it is disable in read side, I think need make these two behavior consistent. If enable, read and writer all need enable in spark.
### Impact
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11730:
URL: https://github.com/apache/hudi/pull/11730#issuecomment-2270568159
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072",
"triggerID" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3aadab3be461a1069a25ccf594de933e44c9abc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11730:
URL: https://github.com/apache/hudi/pull/11730#discussion_r1749517748
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
@KnightChess Instead of changing this, should we flip `public static final boolean DEFAULT_METADATA_ENABLE_FOR_READERS = false` to `true` in `HoodieMetadataConfig`? That controls whether the reader uses metadata table by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11730:
URL: https://github.com/apache/hudi/pull/11730#discussion_r1705064471
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
Did you test the case for reading a metadata-disabled table with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11730:
URL: https://github.com/apache/hudi/pull/11730#discussion_r1758200867
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
A related PR: #7082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11730:
URL: https://github.com/apache/hudi/pull/11730#issuecomment-2271582115
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072",
"triggerID" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"triggerType" : "PUSH"
}, {
"hash" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072",
"triggerID" : "2271492903",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3aadab3be461a1069a25ccf594de933e44c9abc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11730:
URL: https://github.com/apache/hudi/pull/11730#discussion_r1705684118
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
TestHoodieFileIndex I think has test it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #11730:
URL: https://github.com/apache/hudi/pull/11730#issuecomment-2271492903
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11730:
URL: https://github.com/apache/hudi/pull/11730#discussion_r1758562978
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
> You should also verify if flipping DEFAULT_METADATA_ENABLE_FOR_READERS achieves the same.
make sence, but BigQuery use this param too, I only considered spark's scene in the begin. Can flipping it if there is no problem with other engines
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
> You should also verify if flipping DEFAULT_METADATA_ENABLE_FOR_READERS achieves the same.
make sence, but BigQuery use this param too, I only considered spark's scene in the begin. Can flipping it if there is no problem with other engines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11730:
URL: https://github.com/apache/hudi/pull/11730#discussion_r1758565158
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
> A related PR: https://github.com/apache/hudi/pull/7082
close this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess closed pull request #11730: [MINOR] make the read and write behavior consistent in spark
URL: https://github.com/apache/hudi/pull/11730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11730:
URL: https://github.com/apache/hudi/pull/11730#issuecomment-2271564133
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072",
"triggerID" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"triggerType" : "PUSH"
}, {
"hash" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2271492903",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3aadab3be461a1069a25ccf594de933e44c9abc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess closed pull request #11730: [MINOR] make the read and write behavior consistent in spark
URL: https://github.com/apache/hudi/pull/11730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #11730:
URL: https://github.com/apache/hudi/pull/11730#issuecomment-2276555930
iirc metadata is enabled on the reader side by default. Would this PR have MT enabled on the writer side by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11730:
URL: https://github.com/apache/hudi/pull/11730#issuecomment-2270425108
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072",
"triggerID" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3aadab3be461a1069a25ccf594de933e44c9abc6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11730:
URL: https://github.com/apache/hudi/pull/11730#issuecomment-2270416406
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3aadab3be461a1069a25ccf594de933e44c9abc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3aadab3be461a1069a25ccf594de933e44c9abc6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess closed pull request #11730: [MINOR] make the read and write behavior consistent in spark
URL: https://github.com/apache/hudi/pull/11730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] make the read and write behavior consistent in spark [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11730:
URL: https://github.com/apache/hudi/pull/11730#discussion_r1749518585
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -502,6 +502,8 @@ object HoodieFileIndex extends Logging {
val isMetadataTableEnabled = getConfigValue(options, sqlConf, HoodieMetadataConfig.ENABLE.key, null)
if (isMetadataTableEnabled != null) {
properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(isMetadataTableEnabled))
+ } else {
+ properties.setProperty(HoodieMetadataConfig.ENABLE.key(), String.valueOf(HoodieMetadataConfig.ENABLE.defaultValue()))
Review Comment:
You should also verify if flipping `DEFAULT_METADATA_ENABLE_FOR_READERS` achieves the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org