You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2025/01/17 03:08:35 UTC
[PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
cshuo opened a new pull request, #12659:
URL: https://github.com/apache/hudi/pull/12659
### Change Logs
Fix the exception happens when hive scan hudi table after new column is added.
### Impact
Consolidating hive scan to support more schema evolution scenarios.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597533318
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
}, {
"hash" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925",
"triggerID" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a0be89fd6e79531a39125bb4679ba3f4ec47ab7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917)
* db9097873b25d3030045fbc5d3ad7d208263ad40 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597637711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
}, {
"hash" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925",
"triggerID" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"triggerType" : "PUSH"
}, {
"hash" : "d891c0ab1f4a28bf95d98ba82c40ce01753ea90e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d891c0ab1f4a28bf95d98ba82c40ce01753ea90e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db9097873b25d3030045fbc5d3ad7d208263ad40 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925)
* d891c0ab1f4a28bf95d98ba82c40ce01753ea90e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "liucongjy (via GitHub)" <gi...@apache.org>.
liucongjy commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2606214565
> > Is the code you submitted modified based on Hudi1.0? When I tested the submitted code on Hudi 0.15, I still got the same error as before
>
> The main change is in the constructor of `org.apache.hudi.hadoop.avro.HoodieAvroParquetReader`, where the exception happens as you posted in the previous issue, and it's not newly introduced in 1.0. Can you verify that your case reaches this part actually?
The constructor of the HoodieAvroParquetReader class has indeed been executed, and it added a log output. The parameter internalSchemaOption passed to the log is empty, so the current test still encounters the same exception as before. It seems that the InternalSchema returned by the getTableInternalSchemaFromCommitMetadata() method of the TableSchemaResolver class is empty. This indicates that the schema information in the commit file obtained from the Commit Time Line in the Hudi table was not retrieved. However, after adding fields to the table, if data is written back into the table and then queried, the previous exception will no longer be triggered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2606209020
> data must be written into the Hudi table before querying will not throw an error indicating that the added fields do not exist.
make sure schema evolution is enabled, `hoodie.schema.on.read.enable = true`, otherwise alter cmd do not commit table change in hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12659:
URL: https://github.com/apache/hudi/pull/12659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597411681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a0be89fd6e79531a39125bb4679ba3f4ec47ab7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "liucongjy (via GitHub)" <gi...@apache.org>.
liucongjy commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2606166119
I tested it again. After adding fields to the Hudi table, data must be written into the Hudi table before querying will not throw an error indicating that the added fields do not exist. I checked the code and found that during the initialization of the SchemaEvolutionContext class, it did not obtain the CommitsTimeline from HoodieTimeline, which is why the query would throw an error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597531771
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
}, {
"hash" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a0be89fd6e79531a39125bb4679ba3f4ec47ab7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917)
* db9097873b25d3030045fbc5d3ad7d208263ad40 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "liucongjy (via GitHub)" <gi...@apache.org>.
liucongjy commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2606148217
Is the code you submitted modified based on Hudi1.0? When I tested the submitted code on Hudi 0.15, I still got the same error as before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597639779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
}, {
"hash" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925",
"triggerID" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"triggerType" : "PUSH"
}, {
"hash" : "d891c0ab1f4a28bf95d98ba82c40ce01753ea90e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2931",
"triggerID" : "d891c0ab1f4a28bf95d98ba82c40ce01753ea90e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db9097873b25d3030045fbc5d3ad7d208263ad40 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925)
* d891c0ab1f4a28bf95d98ba82c40ce01753ea90e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2931)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2606165029
> 
>
> Is the code you submitted modified based on Hudi1.0? When I tested the submitted code on Hudi 0.15, I still got the same error as before
The main change is in the constructor of `org.apache.hudi.hadoop.avro.HoodieAvroParquetReader`, where the exception happens as you posted in the previous issue, and it's not newly introduced in 1.0. Can you verify that your case reaches this part actually?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "liucongjy (via GitHub)" <gi...@apache.org>.
liucongjy commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2606499006
> > data must be written into the Hudi table before querying will not throw an error indicating that the added fields do not exist.
>
> make sure schema evolution is enabled, `hoodie.schema.on.read.enable = true`, otherwise alter cmd do not commit table change in hudi.
When creating the Hudi table, add the attribute hoodie.schema.on.read.enable = true. The modified code can resolve the previous issue. However, another problem has emerged: the new field values queried by hive sql are out of sync with the partition field values, resulting in the new field values being displayed as the partition field values
<img width="321" alt="image" src="https://github.com/user-attachments/assets/35df3a1f-6e96-40c4-85b8-6bf8ee6ff8b0" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597342133
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a0be89fd6e79531a39125bb4679ba3f4ec47ab7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597343317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a0be89fd6e79531a39125bb4679ba3f4ec47ab7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597631345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
}, {
"hash" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925",
"triggerID" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db9097873b25d3030045fbc5d3ad7d208263ad40 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2597769884
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2917",
"triggerID" : "0a0be89fd6e79531a39125bb4679ba3f4ec47ab7",
"triggerType" : "PUSH"
}, {
"hash" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2925",
"triggerID" : "db9097873b25d3030045fbc5d3ad7d208263ad40",
"triggerType" : "PUSH"
}, {
"hash" : "d891c0ab1f4a28bf95d98ba82c40ce01753ea90e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2931",
"triggerID" : "d891c0ab1f4a28bf95d98ba82c40ce01753ea90e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d891c0ab1f4a28bf95d98ba82c40ce01753ea90e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2931)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2608708951
@liucongjy we'd better track the problem in a new issue or the origin issue, and it would be helpful if you can put more details, e.g., table creating ddl, alter ddl, and select clause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "liucongjy (via GitHub)" <gi...@apache.org>.
liucongjy commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2606216270
> > data must be written into the Hudi table before querying will not throw an error indicating that the added fields do not exist.
>
> make sure schema evolution is enabled, `hoodie.schema.on.read.enable = true`, otherwise alter cmd do not commit table change in hudi.
hoodie.schema.on.read.enable = true This parameter value also adds the log output is true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8880] Fix hive scan exception after new column is added [hudi]
Posted by "FishManHome (via GitHub)" <gi...@apache.org>.
FishManHome commented on PR #12659:
URL: https://github.com/apache/hudi/pull/12659#issuecomment-2808578728
> > > data must be written into the Hudi table before querying will not throw an error indicating that the added fields do not exist.
> >
> >
> > make sure schema evolution is enabled, `hoodie.schema.on.read.enable = true`, otherwise alter cmd do not commit table change in hudi.
>
> When creating the Hudi table, add the attribute hoodie.schema.on.read.enable = true. The modified code can resolve the previous issue. However, another problem has emerged: the new field values queried by hive sql are out of sync with the partition field values, resulting in the new field values being displayed as the partition field values
>
> <img alt="image" width="321" src="https://private-user-images.githubusercontent.com/17332932/405518999-35df3a1f-6e96-40c4-85b8-6bf8ee6ff8b0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDQ3ODY2MTAsIm5iZiI6MTc0NDc4NjMxMCwicGF0aCI6Ii8xNzMzMjkzMi80MDU1MTg5OTktMzVkZjNhMWYtNmU5Ni00MGM0LTg1YjgtNmJmOGVlNmZmOGIwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA0MTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNDE2VDA2NTE1MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWUwZTlhMTY1ZmIwNWY5NDQ1NTA1OWZkY2NiNjIyZGRiNmUzYjE0YWYxMTI1OGE5MGU4ZTI4ODg3NjJmOTE5N2MmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.oKjtntmzGwS1TewPZ0e1mmFEd96TyrvnyymKc1gzHCw">
@liucongjy Expert, I'd like to ask: In the Hudi 0.15.0 version, besides this pull request, what other code changes have you made? I'm currently encountering the same issue as you did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org