You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/11/27 18:37:17 UTC
[PR] [HUDI-8586][WIP] Allow partition stats only for primitive types [hudi]
codope opened a new pull request, #12350:
URL: https://github.com/apache/hudi/pull/12350
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507239894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
}, {
"hash" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083",
"triggerID" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 294983203aa7045f1ea6fe3ae69ccada9c2aec0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081)
* c0f57c15c8ad008ef82aaa70195c433c5bbc9564 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507083885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076)
* 294983203aa7045f1ea6fe3ae69ccada9c2aec0d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12350:
URL: https://github.com/apache/hudi/pull/12350#discussion_r1862585218
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2514,6 +2532,27 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatsRecords(Ho
}
}
+ /**
+ * Given table schema and field to index, checks if field's data type are supported.
+ *
+ * @param columnToIndex column to index
+ * @param tableSchema table schema
+ * @return true if field's data type is supported, false otherwise
+ */
+ public static boolean validateDataTypeForPartitionStats(String columnToIndex, Schema tableSchema) {
+ Schema fieldSchema = getNestedFieldSchemaFromWriteSchema(tableSchema, columnToIndex);
+ // Exclude fields based on logical type
+ if ((fieldSchema.getType() == Schema.Type.INT || fieldSchema.getType() == Schema.Type.LONG)
+ && fieldSchema.getLogicalType() != null) {
+
+ // Skip fields with logical types DATE or TIME_MILLIS for INT, TIMESTAMP_MILLIS for LONG
Review Comment:
how did you arrive at this?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2514,6 +2532,27 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatsRecords(Ho
}
}
+ /**
+ * Given table schema and field to index, checks if field's data type are supported.
+ *
+ * @param columnToIndex column to index
+ * @param tableSchema table schema
+ * @return true if field's data type is supported, false otherwise
+ */
+ public static boolean validateDataTypeForPartitionStats(String columnToIndex, Schema tableSchema) {
Review Comment:
lets add VisibleForTesting annotation. this method does not need to be public
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2389,7 +2395,12 @@ public static HoodieData<HoodieRecord> convertFilesToPartitionStatsRecords(Hoodi
LOG.warn("No columns to index for partition stats index");
return engineContext.emptyHoodieData();
}
- LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndex);
+ // filter columns with only supported types
+ final List<String> validColumnsToIndex = columnsToIndex.stream()
Review Comment:
So, we should include them as part of partition stats as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12350:
URL: https://github.com/apache/hudi/pull/12350#discussion_r1862584099
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2389,7 +2395,12 @@ public static HoodieData<HoodieRecord> convertFilesToPartitionStatsRecords(Hoodi
LOG.warn("No columns to index for partition stats index");
return engineContext.emptyHoodieData();
}
- LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndex);
+ // filter columns with only supported types
+ final List<String> validColumnsToIndex = columnsToIndex.stream()
Review Comment:
as part of this https://github.com/apache/hudi/pull/12310, we are enabling col stats for 3 meta fields (record key, commit time and partition path)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12350:
URL: https://github.com/apache/hudi/pull/12350#discussion_r1862728238
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2389,7 +2395,12 @@ public static HoodieData<HoodieRecord> convertFilesToPartitionStatsRecords(Hoodi
LOG.warn("No columns to index for partition stats index");
return engineContext.emptyHoodieData();
}
- LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndex);
+ // filter columns with only supported types
+ final List<String> validColumnsToIndex = columnsToIndex.stream()
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2506001330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1393a72132290c1dcf69f83092f95a05837e5a44 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507548407
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
}, {
"hash" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083",
"triggerID" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"triggerType" : "PUSH"
}, {
"hash" : "126bde53faa5aabd157a1efc09514947f4c390ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "126bde53faa5aabd157a1efc09514947f4c390ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0f57c15c8ad008ef82aaa70195c433c5bbc9564 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083)
* 126bde53faa5aabd157a1efc09514947f4c390ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #12350:
URL: https://github.com/apache/hudi/pull/12350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507355062
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
}, {
"hash" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083",
"triggerID" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0f57c15c8ad008ef82aaa70195c433c5bbc9564 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507137183
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 294983203aa7045f1ea6fe3ae69ccada9c2aec0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507671929
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
}, {
"hash" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083",
"triggerID" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"triggerType" : "PUSH"
}, {
"hash" : "126bde53faa5aabd157a1efc09514947f4c390ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2089",
"triggerID" : "126bde53faa5aabd157a1efc09514947f4c390ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 126bde53faa5aabd157a1efc09514947f4c390ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2089)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2506004012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1393a72132290c1dcf69f83092f95a05837e5a44 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2506857292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12350:
URL: https://github.com/apache/hudi/pull/12350#discussion_r1862714024
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2514,6 +2532,27 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatsRecords(Ho
}
}
+ /**
+ * Given table schema and field to index, checks if field's data type are supported.
+ *
+ * @param columnToIndex column to index
+ * @param tableSchema table schema
+ * @return true if field's data type is supported, false otherwise
+ */
+ public static boolean validateDataTypeForPartitionStats(String columnToIndex, Schema tableSchema) {
+ Schema fieldSchema = getNestedFieldSchemaFromWriteSchema(tableSchema, columnToIndex);
+ // Exclude fields based on logical type
+ if ((fieldSchema.getType() == Schema.Type.INT || fieldSchema.getType() == Schema.Type.LONG)
+ && fieldSchema.getLogicalType() != null) {
+
+ // Skip fields with logical types DATE or TIME_MILLIS for INT, TIMESTAMP_MILLIS for LONG
Review Comment:
I was checking the timestamop based logical types in avro - https://avro.apache.org/docs/1.11.0/spec.html#Logical+Types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2506812795
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1393a72132290c1dcf69f83092f95a05837e5a44 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057)
* ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507238131
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
}, {
"hash" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 294983203aa7045f1ea6fe3ae69ccada9c2aec0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081)
* c0f57c15c8ad008ef82aaa70195c433c5bbc9564 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2506137365
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1393a72132290c1dcf69f83092f95a05837e5a44 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507082842
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076)
* 294983203aa7045f1ea6fe3ae69ccada9c2aec0d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2506813526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1393a72132290c1dcf69f83092f95a05837e5a44 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057)
* ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8586] Allow partition stats only for primitive types [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12350:
URL: https://github.com/apache/hudi/pull/12350#issuecomment-2507550900
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2057",
"triggerID" : "1393a72132290c1dcf69f83092f95a05837e5a44",
"triggerType" : "PUSH"
}, {
"hash" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2076",
"triggerID" : "ab7e55a9f65af5bfb0a1c094cf8f10a569d193e1",
"triggerType" : "PUSH"
}, {
"hash" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2081",
"triggerID" : "294983203aa7045f1ea6fe3ae69ccada9c2aec0d",
"triggerType" : "PUSH"
}, {
"hash" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083",
"triggerID" : "c0f57c15c8ad008ef82aaa70195c433c5bbc9564",
"triggerType" : "PUSH"
}, {
"hash" : "126bde53faa5aabd157a1efc09514947f4c390ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2089",
"triggerID" : "126bde53faa5aabd157a1efc09514947f4c390ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0f57c15c8ad008ef82aaa70195c433c5bbc9564 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2083)
* 126bde53faa5aabd157a1efc09514947f4c390ff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2089)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org