You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2024/11/21 19:33:54 UTC
[PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
jonvex opened a new pull request, #12310:
URL: https://github.com/apache/hudi/pull/12310
### Change Logs
if user does not specify columns we generate stats for every single column. If they have hundreds or thousands of columns this will have very bad performance.
### Impact
config to limit number of colstats generated
### Risk level (write none, low medium or high below)
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2496239530
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9599cff079c21ca902255d213c63b4201228f52 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906)
* 98ab8b5c927de2f760de245c2303edaf2822f255 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1859452616
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1166,22 +1166,31 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+
+ }
+
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
List<String> columnNames) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames));
+ return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
}
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
Lazy<Option<Schema>> tableSchema) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema));
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), false);
}
private static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
- Either<List<String>, Lazy<Option<Schema>>> tableSchema) {
- checkState(metadataConfig.isColumnStatsIndexEnabled());
+ Either<List<String>, Lazy<Option<Schema>>> tableSchema,
+ boolean overrideEnableCheck) {
+ checkState(overrideEnableCheck || metadataConfig.isColumnStatsIndexEnabled());
Review Comment:
what this overrideEnableCheck ?
we can't have any code in source code just for the purpose testing and not being used in any other production code paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2504931026
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1861198936
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,82 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames) {
+ return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Lazy<Option<Schema>> tableSchema,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), overrideEnableCheck);
+ }
- if (!targetColumnsForColumnStatsIndex.isEmpty()) {
- return targetColumnsForColumnStatsIndex;
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Lazy<Option<Schema>> tableSchema) {
+ return getColumnsToIndex(tableConfig, metadataConfig, tableSchema, false);
+ }
+
+ private static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Either<List<String>, Lazy<Option<Schema>>> tableSchema,
+ boolean overrideEnableCheck) {
+ checkState(overrideEnableCheck || metadataConfig.isColumnStatsIndexEnabled());
+ Stream<String> columnsToIndexWithoutRequiredMetas = getColumnsToIndexWithoutRequiredMetas(metadataConfig, tableSchema);
+ if (!tableConfig.populateMetaFields()) {
+ return columnsToIndexWithoutRequiredMetas.collect(Collectors.toList());
}
- Option<Schema> writerSchemaOpt = lazyWriterSchemaOpt.get();
- return writerSchemaOpt
- .map(writerSchema ->
- writerSchema.getFields().stream()
- .map(Schema.Field::name)
- .collect(Collectors.toList()))
- .orElse(Collections.emptyList());
+ return Stream.concat(Arrays.stream(META_COLS_TO_ALWAYS_INDEX), columnsToIndexWithoutRequiredMetas).collect(Collectors.toList());
+ }
+
+ private static Stream<String> getColumnsToIndexWithoutRequiredMetas(HoodieMetadataConfig metadataConfig, Either<List<String>, Lazy<Option<Schema>>> tableSchema) {
Review Comment:
It still has some meta fields if the user included them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2506770376
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
}, {
"hash" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072",
"triggerID" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b084fdb7b684a86941a2d0106e2564213069146 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051)
* cb2cab446be1c8f6e2269b84b47712962e4ae8f1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2492101309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3fef632b8146cbb20cf8f7734cdbc4f2d39651f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2492457595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1821",
"triggerID" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"triggerType" : "PUSH"
}, {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885b9ff533be8da6c0df730cff351fab332232fa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2504779982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988)
* 7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2498935017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98ab8b5c927de2f760de245c2303edaf2822f255 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908)
* c660040d39c08e0e4258f17dddb46134ae8991b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2506975807
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
}, {
"hash" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072",
"triggerID" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"triggerType" : "PUSH"
}, {
"hash" : "2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2079",
"triggerID" : "2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2cab446be1c8f6e2269b84b47712962e4ae8f1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072)
* 2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2079)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505644705
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043)
* 8b084fdb7b684a86941a2d0106e2564213069146 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2506769504
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
}, {
"hash" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b084fdb7b684a86941a2d0106e2564213069146 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051)
* cb2cab446be1c8f6e2269b84b47712962e4ae8f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2494127133
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885b9ff533be8da6c0df730cff351fab332232fa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2499275733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4e13deacd6f4ec5b79bc1131a6574e82512f606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1857282492
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,51 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
- if (!targetColumnsForColumnStatsIndex.isEmpty()) {
- return targetColumnsForColumnStatsIndex;
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2502440142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2496259075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9599cff079c21ca902255d213c63b4201228f52 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906)
* 98ab8b5c927de2f760de245c2303edaf2822f255 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2502327674
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c5a4d67dffabe5de28498daadf7efdf24887eea Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985)
* a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2494669297
can you update PR desc w/ the new config.
explain how it works when someone explicitly overrides the list of columns to index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2499202892
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c660040d39c08e0e4258f17dddb46134ae8991b8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937)
* f4e13deacd6f4ec5b79bc1131a6574e82512f606 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1854572540
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -173,6 +173,14 @@ public final class HoodieMetadataConfig extends HoodieConfig {
.sinceVersion("0.11.0")
.withDocumentation("Comma-separated list of columns for which column stats index will be built. If not set, all columns will be indexed");
+ public static final ConfigProperty<Integer> COLUMN_STATS_INDEX_MAX_COLUMNS = ConfigProperty
Review Comment:
can we do
```
METADATA_PREFIX + ".index.column.stats.max.columns.to.index"
```
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -413,6 +421,10 @@ public List<String> getColumnsEnabledForColumnStatsIndex() {
return StringUtils.split(getString(COLUMN_STATS_INDEX_FOR_COLUMNS), CONFIG_VALUES_DELIMITER);
}
+ public Integer getMaximumColumnsForColumnStats() {
Review Comment:
once you fix the config key, do change the getter method names accordingly.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1193,20 +1200,20 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
*/
private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
List<String> targetColumnsForColumnStatsIndex,
+ int maxColumnsToIndex,
Lazy<Option<Schema>> lazyWriterSchemaOpt) {
checkState(isColumnStatsIndexEnabled);
if (!targetColumnsForColumnStatsIndex.isEmpty()) {
return targetColumnsForColumnStatsIndex;
+ } else {
+ if (maxColumnsToIndex <= 0) {
+ return Collections.emptyList();
+ }
+ return lazyWriterSchemaOpt.get()
+ .map(tableSchema -> getFirstNFieldNames(tableSchema, maxColumnsToIndex))
Review Comment:
and should we add 5 to the value that user sets to account for meta fields?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2525,4 +2536,31 @@ Map<String, Long> getFileNameToSizeMap() {
return filenameToSizeMap;
}
}
+
+ public static List<String> getColumnsToIndex(HoodieMetadataConfig metadataConfig, Schema tableSchema) {
Review Comment:
why do we have multiple methods to deduce the columns to index?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2525,4 +2536,31 @@ Map<String, Long> getFileNameToSizeMap() {
return filenameToSizeMap;
}
}
+
+ public static List<String> getColumnsToIndex(HoodieMetadataConfig metadataConfig, Schema tableSchema) {
+ List<String> userSetStrings = metadataConfig.getColumnsEnabledForColumnStatsIndex();
Review Comment:
userSetStrings -> userConfiguredColsToIndex
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -367,6 +367,7 @@ public static Map<String, HoodieData<HoodieRecord>> convertMetadataToRecords(Hoo
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex,
+ int maxColsToIndex,
Review Comment:
why can't we deduce the list of columns at higher layer itself.
in other words, why do we need both List<String> targetColumnsForColumnStatsIndex, and int maxColsToIndex.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestColumnStatsIndexWithSQL.scala:
##########
@@ -67,6 +67,29 @@ class TestColumnStatsIndexWithSQL extends ColumnStatIndexTestBase {
verifyFileIndexAndSQLQueries(commonOpts)
}
+ @ParameterizedTest
+ @MethodSource(Array("testMetadataColumnStatsIndexParams"))
+ def testMetadataColumnStatsIndexWithSQLWithLimitedIndexes(testCase: ColumnStatsTestCase): Unit = {
+ val metadataOpts = Map(
+ HoodieMetadataConfig.ENABLE.key -> "true",
+ HoodieMetadataConfig.ENABLE_METADATA_INDEX_COLUMN_STATS.key -> "true",
+ HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS.key() -> "8"
Review Comment:
once we add +5 as above feedback, I guess you have to reduce this value to 3.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1193,20 +1200,20 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
*/
private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
List<String> targetColumnsForColumnStatsIndex,
+ int maxColumnsToIndex,
Lazy<Option<Schema>> lazyWriterSchemaOpt) {
checkState(isColumnStatsIndexEnabled);
if (!targetColumnsForColumnStatsIndex.isEmpty()) {
return targetColumnsForColumnStatsIndex;
+ } else {
+ if (maxColumnsToIndex <= 0) {
+ return Collections.emptyList();
Review Comment:
how can maxColumnsToIndex < 0?
if some user mis-configures to have a value of 0 or less?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2525,4 +2536,31 @@ Map<String, Long> getFileNameToSizeMap() {
return filenameToSizeMap;
}
}
+
+ public static List<String> getColumnsToIndex(HoodieMetadataConfig metadataConfig, Schema tableSchema) {
+ List<String> userSetStrings = metadataConfig.getColumnsEnabledForColumnStatsIndex();
+ if (!userSetStrings.isEmpty()) {
+ return userSetStrings;
+ } else {
+ return getFirstNFieldNames(tableSchema, metadataConfig.getMaximumColumnsForColumnStats());
Review Comment:
again, +5? same comment as above
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestColumnStatsIndexWithSQL.scala:
##########
@@ -204,33 +227,42 @@ class TestColumnStatsIndexWithSQL extends ColumnStatIndexTestBase {
verifyFileIndexAndSQLQueries(commonOpts)
}
- private def setupTable(testCase: ColumnStatsTestCase, metadataOpts: Map[String, String], commonOpts: Map[String, String], shouldValidate: Boolean): Unit = {
+ private def setupTable(testCase: ColumnStatsTestCase, metadataOpts: Map[String, String], commonOpts: Map[String, String],
Review Comment:
can you add another simple test (even parametrize an existing test). where you set numColsToIndex to say 100. expected behavior should be same as not setting any explicit value for the config of interest.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1193,20 +1200,20 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
*/
private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
List<String> targetColumnsForColumnStatsIndex,
+ int maxColumnsToIndex,
Lazy<Option<Schema>> lazyWriterSchemaOpt) {
checkState(isColumnStatsIndexEnabled);
if (!targetColumnsForColumnStatsIndex.isEmpty()) {
return targetColumnsForColumnStatsIndex;
+ } else {
Review Comment:
So, is getColumnsToIndex) at L 1201 is the only method all callers use to deduce the list of fields to index for col stats?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -413,6 +421,10 @@ public List<String> getColumnsEnabledForColumnStatsIndex() {
return StringUtils.split(getString(COLUMN_STATS_INDEX_FOR_COLUMNS), CONFIG_VALUES_DELIMITER);
}
+ public Integer getMaximumColumnsForColumnStats() {
Review Comment:
we could rename this to
maxColumnsToIndexForColStats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2494131969
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885b9ff533be8da6c0df730cff351fab332232fa Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1855701538
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1155,28 +1146,15 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
try {
- Option<Schema> writerSchema =
- Option.ofNullable(commitMetadata.getMetadata(HoodieCommitMetadata.SCHEMA_KEY))
- .flatMap(writerSchemaStr ->
- isNullOrEmpty(writerSchemaStr)
- ? Option.empty()
- : Option.of(new Schema.Parser().parse(writerSchemaStr)));
-
- HoodieTableConfig tableConfig = dataMetaClient.getTableConfig();
-
- // NOTE: Writer schema added to commit metadata will not contain Hudi's metadata fields
- Option<Schema> tableSchema = writerSchema.map(schema ->
- tableConfig.populateMetaFields() ? addMetadataFields(schema) : schema);
-
- List<String> columnsToIndex = getColumnsToIndex(isColumnStatsIndexEnabled, targetColumnsForColumnStatsIndex,
- Lazy.eagerly(tableSchema));
+ List<String> columnsToIndex = getColumnsToIndex(dataMetaClient.getTableConfig(), metadataConfig,
Review Comment:
not all operations might contain schema in the commit metadata.
for eg, delete partition. can you fallback to table schema from table schema resolver if current commit metadata does not have any schema in the extra metadata.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,51 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
- if (!targetColumnsForColumnStatsIndex.isEmpty()) {
- return targetColumnsForColumnStatsIndex;
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
Review Comment:
do we have UT for this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1861199666
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2239,6 +2271,16 @@ private static List<HoodieColumnRangeMetadata<Comparable>> getFileStatsRangeMeta
return readColumnRangeMetadataFrom(partitionPath, fileName, datasetMetaClient, columnsToIndex, maxBufferSize);
}
+ public static Option<Schema> getTableSchemaFromCommitMetadata(HoodieCommitMetadata commitMetadata, HoodieTableConfig tableConfig) {
Review Comment:
I didn't write this I just refactored 2 places using identical code. I can undo the change if it is too confusing for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505107126
Pushed an update to address my own comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2496237034
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a455b8e3d6695d5533c85a4f2aa32181a019ea9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871)
* c9599cff079c21ca902255d213c63b4201228f52 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2495111087
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885b9ff533be8da6c0df730cff351fab332232fa Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823)
* a455b8e3d6695d5533c85a4f2aa32181a019ea9b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2501894375
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66f6647d755f8de01ca42f868cec60fe32cba20d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981)
* 1c5a4d67dffabe5de28498daadf7efdf24887eea UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2506974392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
}, {
"hash" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072",
"triggerID" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"triggerType" : "PUSH"
}, {
"hash" : "2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2cab446be1c8f6e2269b84b47712962e4ae8f1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072)
* 2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505107133
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030)
* c3e919ff07842819c59ebebeb8007ffe22ab4a40 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505828546
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b084fdb7b684a86941a2d0106e2564213069146 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2501870638
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4e13deacd6f4ec5b79bc1131a6574e82512f606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943)
* 66f6647d755f8de01ca42f868cec60fe32cba20d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2496237671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a455b8e3d6695d5533c85a4f2aa32181a019ea9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871)
* c9599cff079c21ca902255d213c63b4201228f52 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2495112601
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885b9ff533be8da6c0df730cff351fab332232fa Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823)
* a455b8e3d6695d5533c85a4f2aa32181a019ea9b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1857281449
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1155,28 +1146,15 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
try {
- Option<Schema> writerSchema =
- Option.ofNullable(commitMetadata.getMetadata(HoodieCommitMetadata.SCHEMA_KEY))
- .flatMap(writerSchemaStr ->
- isNullOrEmpty(writerSchemaStr)
- ? Option.empty()
- : Option.of(new Schema.Parser().parse(writerSchemaStr)));
-
- HoodieTableConfig tableConfig = dataMetaClient.getTableConfig();
-
- // NOTE: Writer schema added to commit metadata will not contain Hudi's metadata fields
- Option<Schema> tableSchema = writerSchema.map(schema ->
- tableConfig.populateMetaFields() ? addMetadataFields(schema) : schema);
-
- List<String> columnsToIndex = getColumnsToIndex(isColumnStatsIndexEnabled, targetColumnsForColumnStatsIndex,
- Lazy.eagerly(tableSchema));
+ List<String> columnsToIndex = getColumnsToIndex(dataMetaClient.getTableConfig(), metadataConfig,
Review Comment:
This is just a refactor of what it was before, because I saw identical code block being used twice.
You can do command-f for
```
Option<Schema> writerSchema =
Option.ofNullable(commitMetadata.getMetadata(HoodieCommitMetadata.SCHEMA_KEY))
.flatMap(writerSchemaStr ->
isNullOrEmpty(writerSchemaStr)
? Option.empty()
: Option.of(new Schema.Parser().parse(writerSchemaStr)));
```
on master and you will get 2 results in the metadata util file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1860041740
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,82 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames) {
+ return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Lazy<Option<Schema>> tableSchema,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), overrideEnableCheck);
+ }
- if (!targetColumnsForColumnStatsIndex.isEmpty()) {
- return targetColumnsForColumnStatsIndex;
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Lazy<Option<Schema>> tableSchema) {
+ return getColumnsToIndex(tableConfig, metadataConfig, tableSchema, false);
+ }
+
+ private static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
Review Comment:
lets add java docs on the method
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,82 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames) {
+ return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Lazy<Option<Schema>> tableSchema,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), overrideEnableCheck);
+ }
- if (!targetColumnsForColumnStatsIndex.isEmpty()) {
- return targetColumnsForColumnStatsIndex;
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
Review Comment:
do we have UTs directly against these getColumnsToIndex methods. I guess we have 3 to 4 diff methods here and we wanted to have tests for all of them
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,82 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames) {
+ return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
Review Comment:
same comment as above. lets ensure we have just the required access specifiers
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2184,10 +2219,7 @@ public static HoodieData<HoodieRecord> convertFilesToPartitionStatsRecords(Hoodi
HoodieTableMetaClient dataTableMetaClient,
Option<Schema> writerSchemaOpt) {
Lazy<Option<Schema>> lazyWriterSchemaOpt = writerSchemaOpt.isPresent() ? Lazy.eagerly(writerSchemaOpt) : Lazy.lazily(() -> tryResolveSchemaForTable(dataTableMetaClient));
- final List<String> columnsToIndex = getColumnsToIndex(
- metadataConfig.isPartitionStatsIndexEnabled(),
- metadataConfig.getColumnsEnabledForColumnStatsIndex(),
- lazyWriterSchemaOpt);
+ final List<String> columnsToIndex = getColumnsToIndex(dataTableMetaClient.getTableConfig(), metadataConfig, lazyWriterSchemaOpt, metadataConfig.isPartitionStatsIndexEnabled());
Review Comment:
whats the relation b/w partitions stats and the overrideEnableCheck?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,82 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames) {
+ return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ }
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Lazy<Option<Schema>> tableSchema,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), overrideEnableCheck);
+ }
- if (!targetColumnsForColumnStatsIndex.isEmpty()) {
- return targetColumnsForColumnStatsIndex;
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Lazy<Option<Schema>> tableSchema) {
+ return getColumnsToIndex(tableConfig, metadataConfig, tableSchema, false);
+ }
+
+ private static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ Either<List<String>, Lazy<Option<Schema>>> tableSchema,
+ boolean overrideEnableCheck) {
+ checkState(overrideEnableCheck || metadataConfig.isColumnStatsIndexEnabled());
+ Stream<String> columnsToIndexWithoutRequiredMetas = getColumnsToIndexWithoutRequiredMetas(metadataConfig, tableSchema);
+ if (!tableConfig.populateMetaFields()) {
+ return columnsToIndexWithoutRequiredMetas.collect(Collectors.toList());
}
- Option<Schema> writerSchemaOpt = lazyWriterSchemaOpt.get();
- return writerSchemaOpt
- .map(writerSchema ->
- writerSchema.getFields().stream()
- .map(Schema.Field::name)
- .collect(Collectors.toList()))
- .orElse(Collections.emptyList());
+ return Stream.concat(Arrays.stream(META_COLS_TO_ALWAYS_INDEX), columnsToIndexWithoutRequiredMetas).collect(Collectors.toList());
+ }
+
+ private static Stream<String> getColumnsToIndexWithoutRequiredMetas(HoodieMetadataConfig metadataConfig, Either<List<String>, Lazy<Option<Schema>>> tableSchema) {
Review Comment:
rename to getColumnsToIndexWithoutMetaFields
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2239,6 +2271,16 @@ private static List<HoodieColumnRangeMetadata<Comparable>> getFileStatsRangeMeta
return readColumnRangeMetadataFrom(partitionPath, fileName, datasetMetaClient, columnsToIndex, maxBufferSize);
}
+ public static Option<Schema> getTableSchemaFromCommitMetadata(HoodieCommitMetadata commitMetadata, HoodieTableConfig tableConfig) {
Review Comment:
again, lets avoid public methods. can you make it package private
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1185,25 +1163,82 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- /**
- * Get the list of columns for the table for column stats indexing
- */
- private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
- List<String> targetColumnsForColumnStatsIndex,
- Lazy<Option<Schema>> lazyWriterSchemaOpt) {
- checkState(isColumnStatsIndexEnabled);
+ public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
Review Comment:
May be most of existing methods does not follow strict access specifiers.
can you make this package private. if you wanna do UT, you can add visible for testing annotation.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1166,22 +1166,31 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
+ public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
+ HoodieMetadataConfig metadataConfig,
+ List<String> columnNames,
+ boolean overrideEnableCheck) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+
+ }
+
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
List<String> columnNames) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames));
+ return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
}
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
Lazy<Option<Schema>> tableSchema) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema));
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), false);
}
private static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
- Either<List<String>, Lazy<Option<Schema>>> tableSchema) {
- checkState(metadataConfig.isColumnStatsIndexEnabled());
+ Either<List<String>, Lazy<Option<Schema>>> tableSchema,
+ boolean overrideEnableCheck) {
+ checkState(overrideEnableCheck || metadataConfig.isColumnStatsIndexEnabled());
Review Comment:
@jonvex : did you follow on this?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1203,7 +1212,21 @@ private static Stream<String> getColumnsToIndexWithoutRequiredMetas(HoodieMetada
}
private static Stream<String> getFirstNFieldNames(Schema tableSchema, int n) {
- return getFirstNFieldNames(tableSchema.getFields().stream().map(Schema.Field::name), n);
+ return getFirstNFieldNames(tableSchema.getFields().stream()
+ .filter(field -> {
+ switch (resolveNullableSchema(field.schema()).getType()) {
Review Comment:
Can we just re-use the same method. may be we can rename the method to, isDataTypeSupportedForColStats(Schema schema, HoodieRecordType recordType)
and use it wherever required.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2239,6 +2271,16 @@ private static List<HoodieColumnRangeMetadata<Comparable>> getFileStatsRangeMeta
return readColumnRangeMetadataFrom(partitionPath, fileName, datasetMetaClient, columnsToIndex, maxBufferSize);
}
+ public static Option<Schema> getTableSchemaFromCommitMetadata(HoodieCommitMetadata commitMetadata, HoodieTableConfig tableConfig) {
Review Comment:
do we have UTs for this
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/metadata/TestHoodieTableMetadataUtil.java:
##########
@@ -355,4 +358,182 @@ public void testValidateDataTypeForSecondaryIndex() {
// Test for complex fields
assertFalse(validateDataTypeForSecondaryIndex(Arrays.asList("arrayField", "mapField", "structField"), schema));
}
+
+ @Test
+ public void testGetColumnsToIndex() {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+
+ //test default
+ HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .build();
+ List<String> colNames = new ArrayList<>();
+ addNColumns(colNames, HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS.defaultValue() + 10);
+ List<String> expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ addNColumns(expected, HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS.defaultValue());
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with table schema < default
+ int tableSchemaSize = HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS.defaultValue() - 10;
+ assertTrue(tableSchemaSize > 0);
+ colNames = new ArrayList<>();
+ addNColumns(colNames, tableSchemaSize);
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ addNColumns(expected, tableSchemaSize);
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with max val < tableSchema
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withMaxColumnsToIndexForColStats(3)
+ .build();
+ colNames = new ArrayList<>();
+ addNColumns(colNames, HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS.defaultValue() + 10);
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ addNColumns(expected, 3);
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with max val > tableSchema
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withMaxColumnsToIndexForColStats(HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS.defaultValue() + 10)
+ .build();
+ colNames = new ArrayList<>();
+ addNColumns(colNames, tableSchemaSize);
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ addNColumns(expected, tableSchemaSize);
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with list of cols
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withColumnStatsIndexForColumns("col_1,col_7,col_11")
+ .build();
+ colNames = new ArrayList<>();
+ addNColumns(colNames, 15);
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ expected.add("col_1");
+ expected.add("col_7");
+ expected.add("col_11");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with list of cols longer than config
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withMaxColumnsToIndexForColStats(1)
+ .withColumnStatsIndexForColumns("col_1,col_7,col_11")
+ .build();
+ colNames = new ArrayList<>();
+ addNColumns(colNames, 15);
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ expected.add("col_1");
+ expected.add("col_7");
+ expected.add("col_11");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with list of cols including meta cols than config
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withColumnStatsIndexForColumns("col_1,col_7,_hoodie_commit_time,col_11,_hoodie_commit_seqno")
+ .build();
+ colNames = new ArrayList<>();
+ addNColumns(colNames, 15);
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ expected.add("col_1");
+ expected.add("col_7");
+ expected.add("col_11");
+ expected.add("_hoodie_commit_seqno");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with avro schema
+ Schema schema = new Schema.Parser().parse(SCHEMA_WITH_AVRO_TYPES);
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withColumnStatsIndexForColumns("booleanField,decimalField,localTimestampMillisField")
+ .build();
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ expected.add("booleanField");
+ expected.add("decimalField");
+ expected.add("localTimestampMillisField");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, Lazy.eagerly(Option.of(schema))));
+
+ //test with avro schema with max cols set
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withMaxColumnsToIndexForColStats(2)
+ .build();
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ expected.add("booleanField");
+ expected.add("intField");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, Lazy.eagerly(Option.of(schema))));
+ //test with avro schema with meta cols
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, Lazy.eagerly(Option.of(HoodieAvroUtils.addMetadataFields(schema)))));
+
+ //test with avro schema with type filter
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withMaxColumnsToIndexForColStats(100)
+ .build();
+ expected = new ArrayList<>(Arrays.asList(HoodieTableMetadataUtil.META_COLS_TO_ALWAYS_INDEX));
+ expected.add("timestamp");
+ expected.add("_row_key");
+ expected.add("partition_path");
+ expected.add("rider");
+ expected.add("driver");
+ expected.add("begin_lat");
+ expected.add("begin_lon");
+ expected.add("end_lat");
+ expected.add("end_lon");
+ expected.add("distance_in_meters");
+ expected.add("seconds_since_epoch");
+ expected.add("weight");
+ expected.add("nation");
+ expected.add("current_date");
+ expected.add("current_ts");
+ expected.add("_hoodie_is_deleted");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, Lazy.eagerly(Option.of(HoodieTestDataGenerator.AVRO_SCHEMA))));
+ //test with avro schema with meta cols
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, Lazy.eagerly(Option.of(HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA)))));
+
+ //test with meta cols disabled
+ tableConfig.setValue(HoodieTableConfig.POPULATE_META_FIELDS.key(), "false");
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .build();
+ colNames = new ArrayList<>();
+ addNColumns(colNames, tableSchemaSize);
+ expected = new ArrayList<>();
+ addNColumns(expected, tableSchemaSize);
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with meta cols disabled with col list
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withColumnStatsIndexForColumns("col_1,col_7,col_11")
+ .build();
+ colNames = new ArrayList<>();
+ addNColumns(colNames, 15);
+ expected = new ArrayList<>();
+ expected.add("col_1");
+ expected.add("col_7");
+ expected.add("col_11");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, colNames));
+
+ //test with meta cols disabled with avro schema
+ metadataConfig = HoodieMetadataConfig.newBuilder()
+ .enable(true).withMetadataIndexColumnStats(true)
+ .withColumnStatsIndexForColumns("booleanField,decimalField,localTimestampMillisField")
+ .build();
+ expected = new ArrayList<>();
+ expected.add("booleanField");
+ expected.add("decimalField");
+ expected.add("localTimestampMillisField");
+ assertEquals(expected, HoodieTableMetadataUtil.getColumnsToIndex(tableConfig, metadataConfig, Lazy.eagerly(Option.of(schema))));
+ }
Review Comment:
looks like we have covered tests here for getColumnsToIndex api. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2499199981
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c660040d39c08e0e4258f17dddb46134ae8991b8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937)
* f4e13deacd6f4ec5b79bc1131a6574e82512f606 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2501865663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4e13deacd6f4ec5b79bc1131a6574e82512f606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943)
* 66f6647d755f8de01ca42f868cec60fe32cba20d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2502051244
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c5a4d67dffabe5de28498daadf7efdf24887eea Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1859447312
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1203,7 +1212,21 @@ private static Stream<String> getColumnsToIndexWithoutRequiredMetas(HoodieMetada
}
private static Stream<String> getFirstNFieldNames(Schema tableSchema, int n) {
- return getFirstNFieldNames(tableSchema.getFields().stream().map(Schema.Field::name), n);
+ return getFirstNFieldNames(tableSchema.getFields().stream()
+ .filter(field -> {
+ switch (resolveNullableSchema(field.schema()).getType()) {
Review Comment:
From what I can guage, these are the supported types
```
private static boolean canCompare(Schema schema, HoodieRecordType recordType) {
// if recordType is SPARK then we cannot compare RECORD and ARRAY types in addition to MAP type
if (recordType == HoodieRecordType.SPARK) {
return schema.getType() != Schema.Type.RECORD && schema.getType() != Schema.Type.ARRAY && schema.getType() != Schema.Type.MAP;
}
return schema.getType() != Schema.Type.MAP;
}
```
ref: https://github.com/apache/hudi/blob/5c28762c80c678e66df662c0f4e4c2855766840f/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java#L1425
Curious to understand from where did you get this list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2501890003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4e13deacd6f4ec5b79bc1131a6574e82512f606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943)
* 66f6647d755f8de01ca42f868cec60fe32cba20d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981)
* 1c5a4d67dffabe5de28498daadf7efdf24887eea UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1861409893
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1163,78 +1176,69 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
- public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
-
- public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
- HoodieMetadataConfig metadataConfig,
- List<String> columnNames,
- boolean overrideEnableCheck) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+ @VisibleForTesting
+ static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ private static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
- }
-
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
List<String> columnNames) {
- return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), Option.empty());
Review Comment:
Punted this for now, since this requires changing the core MDT writer apis to take in recordMerger as an argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505117968
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030)
* c3e919ff07842819c59ebebeb8007ffe22ab4a40 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041)
* 1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2492105148
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1821",
"triggerID" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3fef632b8146cbb20cf8f7734cdbc4f2d39651f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1821)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1854901776
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1193,20 +1200,20 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
*/
private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
List<String> targetColumnsForColumnStatsIndex,
+ int maxColumnsToIndex,
Lazy<Option<Schema>> lazyWriterSchemaOpt) {
checkState(isColumnStatsIndexEnabled);
if (!targetColumnsForColumnStatsIndex.isEmpty()) {
return targetColumnsForColumnStatsIndex;
+ } else {
+ if (maxColumnsToIndex <= 0) {
+ return Collections.emptyList();
+ }
+ return lazyWriterSchemaOpt.get()
+ .map(tableSchema -> getFirstNFieldNames(tableSchema, maxColumnsToIndex))
Review Comment:
So you always want to index all the meta fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1854901182
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -367,6 +367,7 @@ public static Map<String, HoodieData<HoodieRecord>> convertMetadataToRecords(Hoo
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex,
+ int maxColsToIndex,
Review Comment:
We need the schema, and there is stuff with Lazy<Schema> maybe it is an over optimization? If we can get the schema at the higher layer we can just do it there but idk this code well if it's ok to make that change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2501975581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66f6647d755f8de01ca42f868cec60fe32cba20d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981)
* 1c5a4d67dffabe5de28498daadf7efdf24887eea Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2502329904
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c5a4d67dffabe5de28498daadf7efdf24887eea Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985)
* a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2504781867
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988)
* 7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505227265
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2496312637
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98ab8b5c927de2f760de245c2303edaf2822f255 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1861291942
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1163,78 +1176,69 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
- public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
-
- public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
- HoodieMetadataConfig metadataConfig,
- List<String> columnNames,
- boolean overrideEnableCheck) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+ @VisibleForTesting
+ static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ private static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
- }
-
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
List<String> columnNames) {
- return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), Option.empty());
Review Comment:
We could get the recordMerger from HoodieWriteConfig and fetch the record type right?
why set it to Option.empty ?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1163,78 +1176,69 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
}
}
- public static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
- public static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
-
- public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
- HoodieMetadataConfig metadataConfig,
- List<String> columnNames,
- boolean overrideEnableCheck) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), overrideEnableCheck);
+ @VisibleForTesting
+ static final String[] META_COLS_TO_ALWAYS_INDEX = {COMMIT_TIME_METADATA_FIELD, RECORD_KEY_METADATA_FIELD, PARTITION_PATH_METADATA_FIELD};
+ private static final Set<String> META_COL_SET_TO_INDEX = new HashSet<>(Arrays.asList(META_COLS_TO_ALWAYS_INDEX));
- }
-
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
List<String> columnNames) {
- return getColumnsToIndex(tableConfig, metadataConfig, columnNames, false);
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.left(columnNames), Option.empty());
}
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
Lazy<Option<Schema>> tableSchema,
- boolean overrideEnableCheck) {
- return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), overrideEnableCheck);
+ Option<HoodieRecordType> recordType) {
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), recordType);
}
public static List<String> getColumnsToIndex(HoodieTableConfig tableConfig,
HoodieMetadataConfig metadataConfig,
Lazy<Option<Schema>> tableSchema) {
- return getColumnsToIndex(tableConfig, metadataConfig, tableSchema, false);
+ return getColumnsToIndex(tableConfig, metadataConfig, Either.right(tableSchema), Option.empty());
Review Comment:
same here
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1458,8 +1462,12 @@ private static Comparable<?> coerceToComparable(Schema schema, Object val) {
}
private static boolean canCompare(Schema schema, HoodieRecordType recordType) {
+ return isColumnTypeSupported(schema, Option.of(recordType));
+ }
+
+ private static boolean isColumnTypeSupported(Schema schema, Option<HoodieRecordType> recordType) {
Review Comment:
also, if the record type is not set, we should got for the super set.
i.e.
the unsupported types are:
if its AVRO -> MAP
if record type is not set or if its SPARK -> MAP, RECORD, ARRAY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505647825
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043)
* 8b084fdb7b684a86941a2d0106e2564213069146 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #12310:
URL: https://github.com/apache/hudi/pull/12310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2495180652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a455b8e3d6695d5533c85a4f2aa32181a019ea9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505163756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3e919ff07842819c59ebebeb8007ffe22ab4a40 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041)
* 1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2496238926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a455b8e3d6695d5533c85a4f2aa32181a019ea9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871)
* c9599cff079c21ca902255d213c63b4201228f52 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906)
* 98ab8b5c927de2f760de245c2303edaf2822f255 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2499102626
hey @jonvex : looks like there are some test failures. can you follow up. once its green, we are good to land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505119217
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3e919ff07842819c59ebebeb8007ffe22ab4a40 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041)
* 1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2505108410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030)
* c3e919ff07842819c59ebebeb8007ffe22ab4a40 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2506829880
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
}, {
"hash" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072",
"triggerID" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2cab446be1c8f6e2269b84b47712962e4ae8f1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2507032807
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
}, {
"hash" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1871",
"triggerID" : "a455b8e3d6695d5533c85a4f2aa32181a019ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "c9599cff079c21ca902255d213c63b4201228f52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1906",
"triggerID" : "c9599cff079c21ca902255d213c63b4201228f52",
"triggerType" : "PUSH"
}, {
"hash" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1908",
"triggerID" : "98ab8b5c927de2f760de245c2303edaf2822f255",
"triggerType" : "PUSH"
}, {
"hash" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1937",
"triggerID" : "c660040d39c08e0e4258f17dddb46134ae8991b8",
"triggerType" : "PUSH"
}, {
"hash" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1943",
"triggerID" : "f4e13deacd6f4ec5b79bc1131a6574e82512f606",
"triggerType" : "PUSH"
}, {
"hash" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1981",
"triggerID" : "66f6647d755f8de01ca42f868cec60fe32cba20d",
"triggerType" : "PUSH"
}, {
"hash" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1985",
"triggerID" : "1c5a4d67dffabe5de28498daadf7efdf24887eea",
"triggerType" : "PUSH"
}, {
"hash" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1988",
"triggerID" : "a052cf4a6fecb355e8dc6a00e35a6bfb64c4a9c7",
"triggerType" : "PUSH"
}, {
"hash" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2030",
"triggerID" : "7bfe086d6cf47b1a2a65d58de6bbfee89287ae0b",
"triggerType" : "PUSH"
}, {
"hash" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2041",
"triggerID" : "c3e919ff07842819c59ebebeb8007ffe22ab4a40",
"triggerType" : "PUSH"
}, {
"hash" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2043",
"triggerID" : "1b3965ebf1b576ecac8f9f0b49c8dbce7ab1dc67",
"triggerType" : "PUSH"
}, {
"hash" : "8b084fdb7b684a86941a2d0106e2564213069146",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2051",
"triggerID" : "8b084fdb7b684a86941a2d0106e2564213069146",
"triggerType" : "PUSH"
}, {
"hash" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2072",
"triggerID" : "cb2cab446be1c8f6e2269b84b47712962e4ae8f1",
"triggerType" : "PUSH"
}, {
"hash" : "2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2079",
"triggerID" : "2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f7a4b96e3b92bdcfe973cee7f9752aa0e4c9611 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2079)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2492321063
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1821",
"triggerID" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"triggerType" : "PUSH"
}, {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3fef632b8146cbb20cf8f7734cdbc4f2d39651f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1821)
* 885b9ff533be8da6c0df730cff351fab332232fa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12310:
URL: https://github.com/apache/hudi/pull/12310#issuecomment-2492316518
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1821",
"triggerID" : "d3fef632b8146cbb20cf8f7734cdbc4f2d39651f",
"triggerType" : "PUSH"
}, {
"hash" : "885b9ff533be8da6c0df730cff351fab332232fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "885b9ff533be8da6c0df730cff351fab332232fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3fef632b8146cbb20cf8f7734cdbc4f2d39651f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1821)
* 885b9ff533be8da6c0df730cff351fab332232fa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1854909697
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestColumnStatsIndexWithSQL.scala:
##########
@@ -204,33 +227,42 @@ class TestColumnStatsIndexWithSQL extends ColumnStatIndexTestBase {
verifyFileIndexAndSQLQueries(commonOpts)
}
- private def setupTable(testCase: ColumnStatsTestCase, metadataOpts: Map[String, String], commonOpts: Map[String, String], shouldValidate: Boolean): Unit = {
+ private def setupTable(testCase: ColumnStatsTestCase, metadataOpts: Map[String, String], commonOpts: Map[String, String],
Review Comment:
I think every test already does that since the default is 32. What we don't have though is testing if it is set to a low number and we specify the columns to index. I added that into a test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8556] Add config for max number of colstats columns [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12310:
URL: https://github.com/apache/hudi/pull/12310#discussion_r1854901462
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1193,20 +1200,20 @@ public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(Hoodi
*/
private static List<String> getColumnsToIndex(boolean isColumnStatsIndexEnabled,
List<String> targetColumnsForColumnStatsIndex,
+ int maxColumnsToIndex,
Lazy<Option<Schema>> lazyWriterSchemaOpt) {
checkState(isColumnStatsIndexEnabled);
if (!targetColumnsForColumnStatsIndex.isEmpty()) {
return targetColumnsForColumnStatsIndex;
+ } else {
+ if (maxColumnsToIndex <= 0) {
+ return Collections.emptyList();
Review Comment:
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org