You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2023/09/01 01:55:46 UTC
[GitHub] [hudi] Zouxxyy opened a new pull request, #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Zouxxyy opened a new pull request, #9573:
URL: https://github.com/apache/hudi/pull/9573
### Change Logs
Current hive read schema evolution MOR table will error, e.g. :
```sql
-- spark-sql
set hoodie.schema.on.read.enable=true;
create table if not exists hudi_mor_test_tbl (
id bigint,
name string,
num int,
ts bigint,
ds string
) using hudi
tblproperties (
type = 'mor',
primaryKey = 'id',
preCombineField = 'ts'
)
partitioned by (ds);
insert into hudi_mor_test_tbl partition(ds = '20211211') select 1, 'a1', 1000,100;
update hudi_mor_test_tbl set name = 'a2' where id = 1;
alter table hudi_mor_test_tbl rename column name to name_new;
-- hive
select id,name_new from hudi_mor_test_tbl_rt;
Failed with exception java.io.IOException:java.lang.ArrayIndexOutOfBoundsException: 25
```
### Impact
Fix above
### Risk level (write none, low medium or high below)
low
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy closed pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy closed pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
URL: https://github.com/apache/hudi/pull/9573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on PR #9573:
URL: https://github.com/apache/hudi/pull/9573#issuecomment-1698739764
@Zouxxyy
thanks for your fix.
could you pls point out the specific reason for the error? Thank you
i cannot reproduce this problem on hive 3.1.1 (hudi 0.11) with my cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9573:
URL: https://github.com/apache/hudi/pull/9573#issuecomment-1698611596
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ceaaa3e3ec47bb334495d52add0019e9315f8a24",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ceaaa3e3ec47bb334495d52add0019e9315f8a24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ceaaa3e3ec47bb334495d52add0019e9315f8a24 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9573:
URL: https://github.com/apache/hudi/pull/9573#issuecomment-1698622793
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ceaaa3e3ec47bb334495d52add0019e9315f8a24",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19548",
"triggerID" : "ceaaa3e3ec47bb334495d52add0019e9315f8a24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ceaaa3e3ec47bb334495d52add0019e9315f8a24 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19548)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9573:
URL: https://github.com/apache/hudi/pull/9573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9573:
URL: https://github.com/apache/hudi/pull/9573#issuecomment-1698952915
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ceaaa3e3ec47bb334495d52add0019e9315f8a24",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19548",
"triggerID" : "ceaaa3e3ec47bb334495d52add0019e9315f8a24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ceaaa3e3ec47bb334495d52add0019e9315f8a24 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19548)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #9573: [HUDI-6804] Fix hive read schema evolution MOR table
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9573:
URL: https://github.com/apache/hudi/pull/9573#issuecomment-1698802566
> thanks for your fix. could you pls point out the specific reason for the error? Thank you i cannot reproduce this problem on hive 3.1.1 (hudi 0.11) with my cluster
The core change is `internalSchemaOption = Option.of(prunedInternalSchema);` You can remove it and then run the ut add in this patch
hudi 0.11 may not have this patch https://[github.com/apache/hudi/pull/6989](https://github.com/apache/hudi/pull/6989)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org