You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "KnightChess (via GitHub)" <gi...@apache.org> on 2023/12/28 05:35:20 UTC
[PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
KnightChess opened a new pull request, #10427:
URL: https://github.com/apache/hudi/pull/10427
from the picture, csi will use parquet chunk block meta calculate min/max value, and save it to mdt col stat. For complex cols, such as *info array<struct<name: string, age: int>>* , parquet meta will contain only `info.array.name`, `infor.array.age`, but hudi will only calculate `info` column, so this meta in mdt will be null.
And if sql expression contain `IsNotNull(info)`, the file will all be skip.
And consider common cols, which will be add in the future and old file will not contain this col, may cause some other question. So, make code logical clean, Check for null before evaluating the value：min/mav/nullValue.
### Change Logs
- Check for null before evaluating the value：min/mav/nullValue
### Impact
None
### Risk level (write none, low medium or high below)
low
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1870990278
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734",
"triggerID" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f65fd8901d85bc013370abfea7921a6ce88386d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1870998198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734",
"triggerID" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"triggerType" : "PUSH"
}, {
"hash" : "d07bc703721ad554a2ada4c0da1697eb7bd1a996",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d07bc703721ad554a2ada4c0da1697eb7bd1a996",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f65fd8901d85bc013370abfea7921a6ce88386d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734)
* d07bc703721ad554a2ada4c0da1697eb7bd1a996 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1871007708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734",
"triggerID" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"triggerType" : "PUSH"
}, {
"hash" : "d07bc703721ad554a2ada4c0da1697eb7bd1a996",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21741",
"triggerID" : "d07bc703721ad554a2ada4c0da1697eb7bd1a996",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f65fd8901d85bc013370abfea7921a6ce88386d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734)
* d07bc703721ad554a2ada4c0da1697eb7bd1a996 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21741)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1871684141
> I see related changes: #10389
look like met the same problem, close this issue, @danny0405 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess closed pull request #10427: [HUDI-7267]fix dataSkipping with null column stats
URL: https://github.com/apache/hudi/pull/10427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1870889209
I see related changes: https://github.com/apache/hudi/pull/10389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1871125126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734",
"triggerID" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"triggerType" : "PUSH"
}, {
"hash" : "d07bc703721ad554a2ada4c0da1697eb7bd1a996",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21741",
"triggerID" : "d07bc703721ad554a2ada4c0da1697eb7bd1a996",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d07bc703721ad554a2ada4c0da1697eb7bd1a996 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21741)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1870847324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734",
"triggerID" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f65fd8901d85bc013370abfea7921a6ce88386d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21734)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7267]fix dataSkipping with null column stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10427:
URL: https://github.com/apache/hudi/pull/10427#issuecomment-1870843941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f65fd8901d85bc013370abfea7921a6ce88386d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f65fd8901d85bc013370abfea7921a6ce88386d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org