You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/23 01:31:54 UTC
[GitHub] [hudi] yihua opened a new pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
yihua opened a new pull request #4875:
URL: https://github.com/apache/hudi/pull/4875
## What is the purpose of the pull request
This PR fixes the problem of incorrect value and null count read from column stats in metadata table. The root cause is the wrong order of arguments passed to the constructor of `HoodieMetadataColumnStats`. The constructor is:
```
public HoodieMetadataColumnStats(java.lang.String fileName, java.lang.String minValue, java.lang.String maxValue, java.lang.Long valueCount, java.lang.Long nullCount, java.lang.Long totalSize, java.lang.Long totalUncompressedSize, java.lang.Boolean isDeleted) {
```
and the original init is
```
columnStatMetadata = new HoodieMetadataColumnStats(
(String) v.get(COLUMN_STATS_FIELD_RESOURCE_NAME),
(String) v.get(COLUMN_STATS_FIELD_MIN_VALUE),
(String) v.get(COLUMN_STATS_FIELD_MAX_VALUE),
(Long) v.get(COLUMN_STATS_FIELD_NULL_COUNT),
(Long) v.get(COLUMN_STATS_FIELD_VALUE_COUNT),
(Long) v.get(COLUMN_STATS_FIELD_TOTAL_SIZE),
(Long) v.get(COLUMN_STATS_FIELD_TOTAL_UNCOMPRESSED_SIZE),
(Boolean) v.get(COLUMN_STATS_FIELD_IS_DELETED)
);
```
The actual values of `valueCount` and `nullCount` passed in are switched.
(Below screenshot shows the diff of column stats from validation: left: from metadata table, right: from base files (ground truth))
<img width="1906" alt="Screen Shot 2022-02-22 at 16 07 32" src="https://user-images.githubusercontent.com/2497195/155242751-8c12e336-9ab7-42e6-beb9-d9913c8a14a3.png">
## Brief change log
- Uses builder of `HoodieMetadataColumnStats` in `HoodieMetadataPayload` to set each field explicitly.
## Verify this pull request
Runs Hudi Deltastreamer continuous mode writing MOR table, async compaction, cleaner, all multi-modal indexing enabled and verifies that the column stats from metadata table is correct compared to the ground truth from base files.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048426637
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048376974
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048375501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048440826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048417307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048417307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048376974
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
codope merged pull request #4875:
URL: https://github.com/apache/hudi/pull/4875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048375501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048426637
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048425428
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4875: [HUDI-3486] Fix wrong field order for constructing HoodieMetadataColumnStats
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4875:
URL: https://github.com/apache/hudi/pull/4875#issuecomment-1048425428
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f110e33fff004885d41133c76a20e9da0cc36759",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f110e33fff004885d41133c76a20e9da0cc36759",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f110e33fff004885d41133c76a20e9da0cc36759 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org