You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/01/13 00:31:45 UTC
[PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
nsivabalan opened a new pull request, #12623:
URL: https://github.com/apache/hudi/pull/12623
### Change Logs
- Adding support for nested fields with col stats
### Impact
- Adding support for nested fields with col stats
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2609308610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
}, {
"hash" : "25107b6115f6cdb672d20278f690d900aa990082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044",
"triggerID" : "25107b6115f6cdb672d20278f690d900aa990082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25107b6115f6cdb672d20278f690d900aa990082 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12623:
URL: https://github.com/apache/hudi/pull/12623#discussion_r1919531138
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1541,39 +1533,38 @@ private static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
* required meta cols
*
* @param metadataConfig metadata config
- * @param tableSchema either a list of the columns in the table, or a lazy option of the table schema
+ * @param tableSchemaLazyOpt lazy option of the table schema
* @param isTableInitializing true if table is being initialized.
* @param recordType Option of record type. Used to determine which types are valid to index
* @return list of columns that should be indexed
*/
private static Stream<String> getColumnsToIndexWithoutRequiredMetaFields(HoodieMetadataConfig metadataConfig,
- Either<List<String>, Lazy<Option<Schema>>> tableSchema,
+ Lazy<Option<Schema>> tableSchemaLazyOpt,
boolean isTableInitializing,
Option<HoodieRecordType> recordType) {
List<String> columnsToIndex = metadataConfig.getColumnsEnabledForColumnStatsIndex();
if (!columnsToIndex.isEmpty()) {
if (isTableInitializing) {
return columnsToIndex.stream();
}
- // filter for top level fields here.
- List<String> topLevelEligibleFields = tableSchema.isLeft() ? tableSchema.asLeft() :
- (tableSchema.asRight().get().map(schema -> schema.getFields().stream()
- .filter(field -> isColumnTypeSupported(field.schema(), recordType))
- .map(field -> field.name()).collect(toList()))
- .orElse(new ArrayList<String>()));
- return columnsToIndex.stream().filter(fieldName -> !META_COL_SET_TO_INDEX.contains(fieldName) && !fieldName.contains(".")
- && (topLevelEligibleFields.isEmpty() || topLevelEligibleFields.contains(fieldName)));
- }
- if (tableSchema.isLeft()) {
- return getFirstNFieldNames(tableSchema.asLeft().stream(), metadataConfig.maxColumnsToIndexForColStats());
+ // filter for eligible fields
+ Option<Schema> tableSchema = tableSchemaLazyOpt.get();
+ return columnsToIndex.stream().filter(fieldName -> !META_COL_SET_TO_INDEX.contains(fieldName))
+ .filter(fieldName -> {
+ if (tableSchema.isPresent()) {
+ return isColumnTypeSupported(getSchemaForField(tableSchema.get(), fieldName).getValue().schema(), recordType);
+ } else {
+ return true;
Review Comment:
Made minor fixes here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2609050786
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
}, {
"hash" : "25107b6115f6cdb672d20278f690d900aa990082",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044",
"triggerID" : "25107b6115f6cdb672d20278f690d900aa990082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fdd0492920f7fbf321c36abe3d09f142db3c979 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038)
* 25107b6115f6cdb672d20278f690d900aa990082 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2586029472
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cafe676a29d394746502a8a4c0fefd5cfc60b6a0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814)
* 764711ed500b2ce3dd48af1dcf8432c3b1c6458d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608826402
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab9375c4a2fe9d431872812463951acdbb5c5633 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021)
* fff29d13d78f244587f7ac0ec7bd5372310e4a03 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2606269810
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5048705b7725c8de7c665e725a922dcd2055bc0f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922)
* 857feb7d1a58b5cd71c8bab8b339ebd306b397f2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608827801
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab9375c4a2fe9d431872812463951acdbb5c5633 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021)
* fff29d13d78f244587f7ac0ec7bd5372310e4a03 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608077708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 857feb7d1a58b5cd71c8bab8b339ebd306b397f2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012)
* ab9375c4a2fe9d431872812463951acdbb5c5633 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2597416119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 764711ed500b2ce3dd48af1dcf8432c3b1c6458d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815)
* 5048705b7725c8de7c665e725a922dcd2055bc0f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2611320220
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
}, {
"hash" : "25107b6115f6cdb672d20278f690d900aa990082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044",
"triggerID" : "25107b6115f6cdb672d20278f690d900aa990082",
"triggerType" : "PUSH"
}, {
"hash" : "35a7f14df530210b083e051276ceba19014479c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3058",
"triggerID" : "35a7f14df530210b083e051276ceba19014479c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25107b6115f6cdb672d20278f690d900aa990082 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044)
* 35a7f14df530210b083e051276ceba19014479c0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3058)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608080582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 857feb7d1a58b5cd71c8bab8b339ebd306b397f2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012)
* ab9375c4a2fe9d431872812463951acdbb5c5633 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2609047289
@codope : addressed the last couple of feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2586114777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 764711ed500b2ce3dd48af1dcf8432c3b1c6458d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #12623:
URL: https://github.com/apache/hudi/pull/12623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2597415047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 764711ed500b2ce3dd48af1dcf8432c3b1c6458d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815)
* 5048705b7725c8de7c665e725a922dcd2055bc0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608833297
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fff29d13d78f244587f7ac0ec7bd5372310e4a03 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036)
* 5fdd0492920f7fbf321c36abe3d09f142db3c979 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12623:
URL: https://github.com/apache/hudi/pull/12623#discussion_r1926327306
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/schema/HoodieSparkSchemaUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class HoodieSparkSchemaUtils {
Review Comment:
`HoodieSchemaUtils` is also spark-specific and that is why it is in hudi-spark-common module. Also, `HoodieSchemaUtils#checkPartitionSchemaOrder` does handle `StructType`.
Anyway, my main concern is proliferation of utils all across the codebase. We should try to unify all the schema related utils in single class, unless really necessary to split such as when we need to handle spark's struct type which won't be availalble in hudi-common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608890464
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fff29d13d78f244587f7ac0ec7bd5372310e4a03 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036)
* 5fdd0492920f7fbf321c36abe3d09f142db3c979 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2609046014
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fdd0492920f7fbf321c36abe3d09f142db3c979 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12623:
URL: https://github.com/apache/hudi/pull/12623#discussion_r1919801640
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/schema/HoodieSparkSchemaUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class HoodieSparkSchemaUtils {
Review Comment:
In my comment, I did not mean to create a new util class. Just need to move these methods here - https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSchemaUtils.scala
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1551,4 +1552,30 @@ private static Pair<Boolean, String> getIsValueWrapperObfuscated(GenericRecord r
}
return Pair.of(false, null);
}
+
+ /**
+ * Returns field name and the resp data type of the field. The data type will always refer to the leaf node.
+ * for eg, for a.b.c, we turn Pair.of(a.b.c, DataType(c))
+ * @param schema schema of table.
+ * @param fieldName field name of interest.
+ * @return
+ */
+ @VisibleForTesting
+ public static Pair<String, Schema.Field> getSchemaForField(Schema schema, String fieldName) {
+ return getSchemaForField(schema, fieldName, StringUtils.EMPTY_STRING);
+ }
+
+ @VisibleForTesting
+ public static Pair<String, Schema.Field> getSchemaForField(Schema schema, String fieldName, String prefix) {
Review Comment:
Can we please add direct UTs for the methods with simple field, nested field, nullable/non-nullable? I know it's already being tested for nested fields in TestColumnStatsIndex. But, it would be better to cover all cases including edge cases in a separate UT for this methd itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2606402053
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 857feb7d1a58b5cd71c8bab8b339ebd306b397f2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12623:
URL: https://github.com/apache/hudi/pull/12623#discussion_r1925841503
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/schema/HoodieSparkSchemaUtils.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class HoodieSparkSchemaUtils {
Review Comment:
this is spark specific and hence. HoodieSparkUtils deals w/ avro schema, while here we deal w/ spark struct types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608295391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab9375c4a2fe9d431872812463951acdbb5c5633 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2609048369
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
}, {
"hash" : "25107b6115f6cdb672d20278f690d900aa990082",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25107b6115f6cdb672d20278f690d900aa990082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fdd0492920f7fbf321c36abe3d09f142db3c979 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038)
* 25107b6115f6cdb672d20278f690d900aa990082 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2611436689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
}, {
"hash" : "25107b6115f6cdb672d20278f690d900aa990082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044",
"triggerID" : "25107b6115f6cdb672d20278f690d900aa990082",
"triggerType" : "PUSH"
}, {
"hash" : "35a7f14df530210b083e051276ceba19014479c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3058",
"triggerID" : "35a7f14df530210b083e051276ceba19014479c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35a7f14df530210b083e051276ceba19014479c0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3058)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2586001228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cafe676a29d394746502a8a4c0fefd5cfc60b6a0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814)
* 764711ed500b2ce3dd48af1dcf8432c3b1c6458d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2585994093
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cafe676a29d394746502a8a4c0fefd5cfc60b6a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2597414222
@codope : addressed feedback. you can TAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2597545927
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5048705b7725c8de7c665e725a922dcd2055bc0f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12623:
URL: https://github.com/apache/hudi/pull/12623#discussion_r1917854534
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -438,9 +438,8 @@ private void processAppendResult(AppendResult result, List<HoodieRecord> recordL
.getColumnsToIndex(hoodieTable.getMetaClient().getTableConfig(),
config.getMetadataConfig(), Lazy.eagerly(Option.of(writeSchemaWithMetaFields)),
Option.of(this.recordMerger.getRecordType())));
- final List<Schema.Field> fieldsToIndex = writeSchemaWithMetaFields.getFields().stream()
- .filter(field -> columnsToIndexSet.contains(field.name()))
- .collect(Collectors.toList());
+ final List<Pair<String, Schema.Field>> fieldsToIndex = columnsToIndexSet.stream()
Review Comment:
So, the fieldname in the pair looks like `a.b.c` for nested field and the value i.e. schema is only for child `c` or is it the whole schema for `a.b.c`?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -242,18 +241,18 @@ class ColumnStats {
records.forEach((record) -> {
// For each column (field) we have to index update corresponding column stats
// with the values from this record
- targetFields.forEach(field -> {
- ColumnStats colStats = allColumnStats.computeIfAbsent(field.name(), ignored -> new ColumnStats());
- Schema fieldSchema = getNestedFieldSchemaFromWriteSchema(recordSchema, field.name());
+ targetFields.forEach(fieldNameFieldPair -> {
+ ColumnStats colStats = allColumnStats.computeIfAbsent(fieldNameFieldPair.getKey(), ignored -> new ColumnStats());
+ Schema fieldSchema = fieldNameFieldPair.getValue().schema();
Review Comment:
for readability - we can extract pair's key and value to separate fieldName and fieldSchema variables and use that below everywhere.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1541,39 +1533,38 @@ private static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
* required meta cols
*
* @param metadataConfig metadata config
- * @param tableSchema either a list of the columns in the table, or a lazy option of the table schema
+ * @param tableSchemaLazyOpt lazy option of the table schema
* @param isTableInitializing true if table is being initialized.
* @param recordType Option of record type. Used to determine which types are valid to index
* @return list of columns that should be indexed
*/
private static Stream<String> getColumnsToIndexWithoutRequiredMetaFields(HoodieMetadataConfig metadataConfig,
- Either<List<String>, Lazy<Option<Schema>>> tableSchema,
+ Lazy<Option<Schema>> tableSchemaLazyOpt,
boolean isTableInitializing,
Option<HoodieRecordType> recordType) {
List<String> columnsToIndex = metadataConfig.getColumnsEnabledForColumnStatsIndex();
if (!columnsToIndex.isEmpty()) {
if (isTableInitializing) {
return columnsToIndex.stream();
}
- // filter for top level fields here.
- List<String> topLevelEligibleFields = tableSchema.isLeft() ? tableSchema.asLeft() :
- (tableSchema.asRight().get().map(schema -> schema.getFields().stream()
- .filter(field -> isColumnTypeSupported(field.schema(), recordType))
- .map(field -> field.name()).collect(toList()))
- .orElse(new ArrayList<String>()));
- return columnsToIndex.stream().filter(fieldName -> !META_COL_SET_TO_INDEX.contains(fieldName) && !fieldName.contains(".")
- && (topLevelEligibleFields.isEmpty() || topLevelEligibleFields.contains(fieldName)));
- }
- if (tableSchema.isLeft()) {
- return getFirstNFieldNames(tableSchema.asLeft().stream(), metadataConfig.maxColumnsToIndexForColStats());
+ // filter for eligible fields
+ Option<Schema> tableSchema = tableSchemaLazyOpt.get();
+ return columnsToIndex.stream().filter(fieldName -> !META_COL_SET_TO_INDEX.contains(fieldName))
+ .filter(fieldName -> {
+ if (tableSchema.isPresent()) {
+ return isColumnTypeSupported(getSchemaForField(tableSchema.get(), fieldName).getValue().schema(), recordType);
+ } else {
+ return true;
Review Comment:
why return true here?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1586,6 +1577,25 @@ private static Stream<String> getFirstNFieldNames(Stream<String> fieldNames, int
return fieldNames.filter(fieldName -> !HOODIE_META_COLUMNS_WITH_OPERATION.contains(fieldName)).limit(n);
}
+ @VisibleForTesting
+ public static Pair<String, Schema.Field> getSchemaForField(Schema schema, String fieldName) {
+ return getSchemaForField(schema, fieldName, StringUtils.EMPTY_STRING);
+ }
+
+ @VisibleForTesting
+ public static Pair<String, Schema.Field> getSchemaForField(Schema schema, String fieldName, String prefix) {
Review Comment:
Let's move these util methods to `HoodieAvroUtils` in hudi-common. And, can we add some direct UTs for the methods with simple field, nested field, nullable/non-nullable.
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java:
##########
@@ -69,6 +73,25 @@ public class DataSourceUtils {
private static final Logger LOG = LoggerFactory.getLogger(DataSourceUtils.class);
+ @VisibleForTesting
+ public static Pair<String, StructField> getSchemaForField(StructType schema, String fieldName) {
+ return getSchemaForField(schema, fieldName, StringUtils.EMPTY_STRING);
+ }
+
+ @VisibleForTesting
+ public static Pair<String, StructField> getSchemaForField(StructType schema, String fieldName, String prefix) {
Review Comment:
let's move these to `HoodieSchemaUtils` in hudi-spark-common
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/ColumnStatIndexTestBase.scala:
##########
@@ -101,7 +103,12 @@ class ColumnStatIndexTestBase extends HoodieSparkClientTestBase {
val inputDF = if (addNestedFiled) {
preInputDF.withColumn("c9",
- functions.struct("c2").withField("car_brand", lit("abc_brand")))
+ functions.struct("c2").withField("c9_1_car_brand", lit("abc_brand")))
+ .withColumn("c10",
+ functions.struct("c2").withField("c10_1_car_brand", lit("abc_brand"))
+ .withField("c10_1", functions.struct("c2")
+ .withField("c10_2_1_nested_lvl2_field1", lit("random_val1"))
+ .withField("c10_2_1_nested_lvl2_field2", lit("random_val2"))))
Review Comment:
+1 on adding 2 levels of nesting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12623:
URL: https://github.com/apache/hudi/pull/12623#discussion_r1918874397
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -438,9 +438,8 @@ private void processAppendResult(AppendResult result, List<HoodieRecord> recordL
.getColumnsToIndex(hoodieTable.getMetaClient().getTableConfig(),
config.getMetadataConfig(), Lazy.eagerly(Option.of(writeSchemaWithMetaFields)),
Option.of(this.recordMerger.getRecordType())));
- final List<Schema.Field> fieldsToIndex = writeSchemaWithMetaFields.getFields().stream()
- .filter(field -> columnsToIndexSet.contains(field.name()))
- .collect(Collectors.toList());
+ final List<Pair<String, Schema.Field>> fieldsToIndex = columnsToIndexSet.stream()
Review Comment:
yes, key will be "a.b.c" and schema.field will refer to the datatype of "c" since thats what we might need to know. a: what col name to use while encoding data to col stats partition. and the schema field for leaf field ("c" in this context).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2606268261
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5048705b7725c8de7c665e725a922dcd2055bc0f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922)
* 857feb7d1a58b5cd71c8bab8b339ebd306b397f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2585993005
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cafe676a29d394746502a8a4c0fefd5cfc60b6a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2586000193
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cafe676a29d394746502a8a4c0fefd5cfc60b6a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814)
* 764711ed500b2ce3dd48af1dcf8432c3b1c6458d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2611318779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3038",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
}, {
"hash" : "25107b6115f6cdb672d20278f690d900aa990082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044",
"triggerID" : "25107b6115f6cdb672d20278f690d900aa990082",
"triggerType" : "PUSH"
}, {
"hash" : "35a7f14df530210b083e051276ceba19014479c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "35a7f14df530210b083e051276ceba19014479c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25107b6115f6cdb672d20278f690d900aa990082 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3044)
* 35a7f14df530210b083e051276ceba19014479c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12623:
URL: https://github.com/apache/hudi/pull/12623#issuecomment-2608831943
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2814",
"triggerID" : "cafe676a29d394746502a8a4c0fefd5cfc60b6a0",
"triggerType" : "PUSH"
}, {
"hash" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2815",
"triggerID" : "764711ed500b2ce3dd48af1dcf8432c3b1c6458d",
"triggerType" : "PUSH"
}, {
"hash" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2922",
"triggerID" : "5048705b7725c8de7c665e725a922dcd2055bc0f",
"triggerType" : "PUSH"
}, {
"hash" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3012",
"triggerID" : "857feb7d1a58b5cd71c8bab8b339ebd306b397f2",
"triggerType" : "PUSH"
}, {
"hash" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021",
"triggerID" : "ab9375c4a2fe9d431872812463951acdbb5c5633",
"triggerType" : "PUSH"
}, {
"hash" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036",
"triggerID" : "fff29d13d78f244587f7ac0ec7bd5372310e4a03",
"triggerType" : "PUSH"
}, {
"hash" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fdd0492920f7fbf321c36abe3d09f142db3c979",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab9375c4a2fe9d431872812463951acdbb5c5633 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3021)
* fff29d13d78f244587f7ac0ec7bd5372310e4a03 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3036)
* 5fdd0492920f7fbf321c36abe3d09f142db3c979 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8582] Adding support for nested fields with col stats [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12623:
URL: https://github.com/apache/hudi/pull/12623#discussion_r1926342588
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSchemaUtils.scala:
##########
@@ -45,6 +45,8 @@ import scala.collection.JavaConverters._
object HoodieSchemaUtils {
private val log = LoggerFactory.getLogger(getClass)
+
+
Review Comment:
nit: please remove the unnecessary newline. Also, let's move the schema util methods in `HoodieSparkSchemaUtils` to this object - see https://github.com/apache/hudi/pull/12623/files#r1926327306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org