You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2025/08/20 19:06:41 UTC
[PR] [HUDI-9731] Add index definition for Record level Index [hudi]
lokeshj1703 opened a new pull request, #13741:
URL: https://github.com/apache/hudi/pull/13741
### Change Logs
Currently we do not create index definition for record index. We can create an index definition and store whether the index is partitioned or not.
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3207721045
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5ff782dfd82d056badb74123ade5c4c127f8421 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3207704944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5ff782dfd82d056badb74123ade5c4c127f8421 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3213770515
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2c4ce896ab2c079a41d2660a95649593be84df7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3213701750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5ff782dfd82d056badb74123ade5c4c127f8421 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986)
* c2c4ce896ab2c079a41d2660a95649593be84df7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex merged PR #13741:
URL: https://github.com/apache/hudi/pull/13741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13741:
URL: https://github.com/apache/hudi/pull/13741#discussion_r2348685260
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -656,16 +656,7 @@ public static <R> HoodieRecord<R> createNewTaggedHoodieRecord(HoodieRecord<R> ol
public static void register(HoodieTableMetaClient metaClient, HoodieIndexDefinition indexDefinition) {
LOG.info("Registering index {} of using {}", indexDefinition.getIndexName(), indexDefinition.getIndexType());
// build HoodieIndexMetadata and then add to index definition file
- boolean indexDefnUpdated = metaClient.buildIndexDefinition(indexDefinition);
- if (indexDefnUpdated) {
- String indexMetaPath = metaClient.getIndexDefinitionPath();
- // update table config if necessary
- if (!metaClient.getTableConfig().getProps().containsKey(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH.key())
- || !metaClient.getTableConfig().getRelativeIndexDefinitionPath().isPresent()) {
- metaClient.getTableConfig().setValue(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH, FSUtils.getRelativePartitionPath(metaClient.getBasePath(), new StoragePath(indexMetaPath)));
- HoodieTableConfig.update(metaClient.getStorage(), metaClient.getMetaPath(), metaClient.getTableConfig().getProps());
- }
- }
+ metaClient.buildIndexDefinition(indexDefinition);
Review Comment:
Addressed. Currently we are not using its return value but it might be useful in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3291877552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
}, {
"hash" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"triggerType" : "PUSH"
}, {
"hash" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060",
"triggerID" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"triggerType" : "PUSH"
}, {
"hash" : "0c91714fcecf6cd348c73d59aec5c5963c690403",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c91714fcecf6cd348c73d59aec5c5963c690403",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb90a24a3fe161580c467ce14a73e4e7d09f1edc UNKNOWN
* a42c3c943fa2d31567c56354dc74158968c2b695 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060)
* 0c91714fcecf6cd348c73d59aec5c5963c690403 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3207713626
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5ff782dfd82d056badb74123ade5c4c127f8421 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3214644239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
}, {
"hash" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"triggerType" : "PUSH"
}, {
"hash" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4b48b8b81c1456289bdfc9df759a8ab4e9b649d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058)
* cb90a24a3fe161580c467ce14a73e4e7d09f1edc UNKNOWN
* a42c3c943fa2d31567c56354dc74158968c2b695 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13741:
URL: https://github.com/apache/hudi/pull/13741#discussion_r2348808352
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -738,8 +742,8 @@ void initializeFilegroupsAndCommit(MetadataPartitionType partitionType,
}
private void initializeFilegroupsAndCommitToRecordIndexPartition(String commitTimeForPartition,
- Lazy<List<Pair<String, FileSlice>>> lazyLatestMergedPartitionFileSliceList) throws IOException {
- if (dataWriteConfig.isPartitionedRecordIndexEnabled()) {
+ Lazy<List<Pair<String, FileSlice>>> lazyLatestMergedPartitionFileSliceList, boolean isPartitioned) throws IOException {
Review Comment:
Addressed
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -476,7 +478,9 @@ private boolean initializeFromFilesystem(String dataTableInstantTime, List<Metad
initializeFilegroupsAndCommit(partitionType, COLUMN_STATS.getPartitionPath(), fileGroupCountAndRecordsPair, instantTimeForPartition, colStatsColumnsAndRecord.getKey());
break;
case RECORD_INDEX:
- initializeFilegroupsAndCommitToRecordIndexPartition(instantTimeForPartition, lazyLatestMergedPartitionFileSliceList);
+ boolean isPartitioned = dataWriteConfig.isPartitionedRecordIndexEnabled();
+ createRecordIndexDefinition(dataMetaClient, Collections.singletonMap(HoodieRecordIndex.IS_PARTITIONED_OPTION, String.valueOf(isPartitioned)));
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndex.scala:
##########
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RecordLevelIndexTestBase.scala:
##########
Review Comment:
Addressed
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/TestHoodieTableMetaClient.java:
##########
@@ -276,7 +276,7 @@ void testDeleteDefinition() throws IOException {
assertTrue(metaClient.getIndexForMetadataPartition(indexName).isPresent());
assertTrue(metaClient.getStorage().exists(new StoragePath(metaClient.getIndexDefinitionPath())));
metaClient.deleteIndexDefinition(indexName);
- assertTrue(metaClient.getIndexMetadata().isEmpty());
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13741:
URL: https://github.com/apache/hudi/pull/13741#discussion_r2348808900
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java:
##########
@@ -194,13 +200,13 @@ public void drop(HoodieTableMetaClient metaClient, String indexName, boolean ign
Option<HoodieIndexDefinition> indexDefinitionOpt = metaClient.getIndexMetadata()
.map(HoodieIndexMetadata::getIndexDefinitions)
.map(definition -> definition.get(indexName));
- try (SparkRDDWriteClient writeClient = getWriteClient(metaClient, indexDefinitionOpt, Option.empty())) {
+ try (SparkRDDWriteClient writeClient = getWriteClient(metaClient, indexDefinitionOpt, Option.empty(), Collections.emptyMap())) {
writeClient.dropIndex(Collections.singletonList(indexName));
}
}
private SparkRDDWriteClient getWriteClient(HoodieTableMetaClient metaClient, Option<HoodieIndexDefinition> indexDefinitionOpt,
- Option<String> indexTypeOpt) {
+ Option<String> indexTypeOpt, Map<String, String> overrideConfigs) {
Review Comment:
Addressed
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java:
##########
@@ -113,9 +115,13 @@ private void createRecordIndex(HoodieTableMetaClient metaClient, String userInde
throw new HoodieMetadataIndexException("Index already exists: " + userIndexName);
}
+ Map<String, String> overrideOpts = new HashMap<>();
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3214502958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4b48b8b81c1456289bdfc9df759a8ab4e9b649d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3213709015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5ff782dfd82d056badb74123ade5c4c127f8421 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986)
* c2c4ce896ab2c079a41d2660a95649593be84df7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3214543580
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
}, {
"hash" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4b48b8b81c1456289bdfc9df759a8ab4e9b649d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058)
* cb90a24a3fe161580c467ce14a73e4e7d09f1edc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3214490330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2c4ce896ab2c079a41d2660a95649593be84df7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048)
* f4b48b8b81c1456289bdfc9df759a8ab4e9b649d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #13741:
URL: https://github.com/apache/hudi/pull/13741#discussion_r2298296150
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java:
##########
@@ -113,9 +115,13 @@ private void createRecordIndex(HoodieTableMetaClient metaClient, String userInde
throw new HoodieMetadataIndexException("Index already exists: " + userIndexName);
}
+ Map<String, String> overrideOpts = new HashMap<>();
Review Comment:
Use singletonlist and emptylist instead
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndex.scala:
##########
Review Comment:
Also add some validations to `TestPartitionedRecordLevelIndex`
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/TestHoodieTableMetaClient.java:
##########
@@ -276,7 +276,7 @@ void testDeleteDefinition() throws IOException {
assertTrue(metaClient.getIndexForMetadataPartition(indexName).isPresent());
assertTrue(metaClient.getStorage().exists(new StoragePath(metaClient.getIndexDefinitionPath())));
metaClient.deleteIndexDefinition(indexName);
- assertTrue(metaClient.getIndexMetadata().isEmpty());
Review Comment:
assertTrue(metaClient.getIndexMetadata().isPresent());
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -656,16 +656,7 @@ public static <R> HoodieRecord<R> createNewTaggedHoodieRecord(HoodieRecord<R> ol
public static void register(HoodieTableMetaClient metaClient, HoodieIndexDefinition indexDefinition) {
LOG.info("Registering index {} of using {}", indexDefinition.getIndexName(), indexDefinition.getIndexType());
// build HoodieIndexMetadata and then add to index definition file
- boolean indexDefnUpdated = metaClient.buildIndexDefinition(indexDefinition);
- if (indexDefnUpdated) {
- String indexMetaPath = metaClient.getIndexDefinitionPath();
- // update table config if necessary
- if (!metaClient.getTableConfig().getProps().containsKey(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH.key())
- || !metaClient.getTableConfig().getRelativeIndexDefinitionPath().isPresent()) {
- metaClient.getTableConfig().setValue(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH, FSUtils.getRelativePartitionPath(metaClient.getBasePath(), new StoragePath(indexMetaPath)));
- HoodieTableConfig.update(metaClient.getStorage(), metaClient.getMetaPath(), metaClient.getTableConfig().getProps());
- }
- }
+ metaClient.buildIndexDefinition(indexDefinition);
Review Comment:
If we get rid of everything here, then we should just get rid of the static method entirely because all it is doing is invoking this on the meta client
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java:
##########
@@ -194,13 +200,13 @@ public void drop(HoodieTableMetaClient metaClient, String indexName, boolean ign
Option<HoodieIndexDefinition> indexDefinitionOpt = metaClient.getIndexMetadata()
.map(HoodieIndexMetadata::getIndexDefinitions)
.map(definition -> definition.get(indexName));
- try (SparkRDDWriteClient writeClient = getWriteClient(metaClient, indexDefinitionOpt, Option.empty())) {
+ try (SparkRDDWriteClient writeClient = getWriteClient(metaClient, indexDefinitionOpt, Option.empty(), Collections.emptyMap())) {
writeClient.dropIndex(Collections.singletonList(indexName));
}
}
private SparkRDDWriteClient getWriteClient(HoodieTableMetaClient metaClient, Option<HoodieIndexDefinition> indexDefinitionOpt,
- Option<String> indexTypeOpt) {
+ Option<String> indexTypeOpt, Map<String, String> overrideConfigs) {
Review Comment:
no need for "override" in the name
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RecordLevelIndexTestBase.scala:
##########
Review Comment:
revert changes to this file
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -259,6 +260,13 @@ public boolean buildIndexDefinition(HoodieIndexDefinition indexDefinition) {
}
if (updateIndexDefn) {
writeIndexMetadataToStorage();
+ String indexMetaPath = getIndexDefinitionPath();
+ // update table config if necessary
Review Comment:
why wasn't this needed before?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -476,7 +478,9 @@ private boolean initializeFromFilesystem(String dataTableInstantTime, List<Metad
initializeFilegroupsAndCommit(partitionType, COLUMN_STATS.getPartitionPath(), fileGroupCountAndRecordsPair, instantTimeForPartition, colStatsColumnsAndRecord.getKey());
break;
case RECORD_INDEX:
- initializeFilegroupsAndCommitToRecordIndexPartition(instantTimeForPartition, lazyLatestMergedPartitionFileSliceList);
+ boolean isPartitioned = dataWriteConfig.isPartitionedRecordIndexEnabled();
+ createRecordIndexDefinition(dataMetaClient, Collections.singletonMap(HoodieRecordIndex.IS_PARTITIONED_OPTION, String.valueOf(isPartitioned)));
Review Comment:
can we pull this into `initializeFilegroupsAndCommitToRecordIndexPartition`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3291910886
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
}, {
"hash" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"triggerType" : "PUSH"
}, {
"hash" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060",
"triggerID" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"triggerType" : "PUSH"
}, {
"hash" : "0c91714fcecf6cd348c73d59aec5c5963c690403",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8535",
"triggerID" : "0c91714fcecf6cd348c73d59aec5c5963c690403",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb90a24a3fe161580c467ce14a73e4e7d09f1edc UNKNOWN
* a42c3c943fa2d31567c56354dc74158968c2b695 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060)
* 0c91714fcecf6cd348c73d59aec5c5963c690403 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8535)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3214482010
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2c4ce896ab2c079a41d2660a95649593be84df7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048)
* f4b48b8b81c1456289bdfc9df759a8ab4e9b649d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3214697711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
}, {
"hash" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"triggerType" : "PUSH"
}, {
"hash" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060",
"triggerID" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4b48b8b81c1456289bdfc9df759a8ab4e9b649d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058)
* cb90a24a3fe161580c467ce14a73e4e7d09f1edc UNKNOWN
* a42c3c943fa2d31567c56354dc74158968c2b695 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3214922677
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
}, {
"hash" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"triggerType" : "PUSH"
}, {
"hash" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060",
"triggerID" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb90a24a3fe161580c467ce14a73e4e7d09f1edc UNKNOWN
* a42c3c943fa2d31567c56354dc74158968c2b695 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13741:
URL: https://github.com/apache/hudi/pull/13741#discussion_r2299289211
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -44,6 +46,28 @@ import scala.util.Using
class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val sqlTempTable = "tbl"
+ @ParameterizedTest
+ @ValueSource(booleans = Array(true, false))
+ def testRLICreationUsingSQL(isPartitioned: Boolean): Unit = {
Review Comment:
why do we need a new test. why can't we add validations to one of the existing test only?
`TestPartitionedRecordLevelIndex`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -738,8 +742,8 @@ void initializeFilegroupsAndCommit(MetadataPartitionType partitionType,
}
private void initializeFilegroupsAndCommitToRecordIndexPartition(String commitTimeForPartition,
- Lazy<List<Pair<String, FileSlice>>> lazyLatestMergedPartitionFileSliceList) throws IOException {
- if (dataWriteConfig.isPartitionedRecordIndexEnabled()) {
+ Lazy<List<Pair<String, FileSlice>>> lazyLatestMergedPartitionFileSliceList, boolean isPartitioned) throws IOException {
Review Comment:
`isPartitionedRli`. if not, might get confused with data table partitions
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -259,6 +260,13 @@ public boolean buildIndexDefinition(HoodieIndexDefinition indexDefinition) {
}
if (updateIndexDefn) {
writeIndexMetadataToStorage();
+ String indexMetaPath = getIndexDefinitionPath();
+ // update table config if necessary
Review Comment:
He moved this code from HoodieIndexUtils
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -476,7 +478,9 @@ private boolean initializeFromFilesystem(String dataTableInstantTime, List<Metad
initializeFilegroupsAndCommit(partitionType, COLUMN_STATS.getPartitionPath(), fileGroupCountAndRecordsPair, instantTimeForPartition, colStatsColumnsAndRecord.getKey());
break;
case RECORD_INDEX:
- initializeFilegroupsAndCommitToRecordIndexPartition(instantTimeForPartition, lazyLatestMergedPartitionFileSliceList);
+ boolean isPartitioned = dataWriteConfig.isPartitionedRecordIndexEnabled();
+ createRecordIndexDefinition(dataMetaClient, Collections.singletonMap(HoodieRecordIndex.IS_PARTITIONED_OPTION, String.valueOf(isPartitioned)));
Review Comment:
+1
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -656,16 +656,7 @@ public static <R> HoodieRecord<R> createNewTaggedHoodieRecord(HoodieRecord<R> ol
public static void register(HoodieTableMetaClient metaClient, HoodieIndexDefinition indexDefinition) {
LOG.info("Registering index {} of using {}", indexDefinition.getIndexName(), indexDefinition.getIndexType());
// build HoodieIndexMetadata and then add to index definition file
- boolean indexDefnUpdated = metaClient.buildIndexDefinition(indexDefinition);
- if (indexDefnUpdated) {
- String indexMetaPath = metaClient.getIndexDefinitionPath();
- // update table config if necessary
- if (!metaClient.getTableConfig().getProps().containsKey(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH.key())
- || !metaClient.getTableConfig().getRelativeIndexDefinitionPath().isPresent()) {
- metaClient.getTableConfig().setValue(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH, FSUtils.getRelativePartitionPath(metaClient.getBasePath(), new StoragePath(indexMetaPath)));
- HoodieTableConfig.update(metaClient.getStorage(), metaClient.getMetaPath(), metaClient.getTableConfig().getProps());
- }
- }
+ metaClient.buildIndexDefinition(indexDefinition);
Review Comment:
Also, shoudln't we make `metaclient.buildIndexDefinition` return void?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13741:
URL: https://github.com/apache/hudi/pull/13741#discussion_r2348807816
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -44,6 +46,28 @@ import scala.util.Using
class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val sqlTempTable = "tbl"
+ @ParameterizedTest
+ @ValueSource(booleans = Array(true, false))
+ def testRLICreationUsingSQL(isPartitioned: Boolean): Unit = {
Review Comment:
This type of test does not exist in RLI test suite. We are directly calling the index creation APIs here.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -44,6 +46,28 @@ import scala.util.Using
class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val sqlTempTable = "tbl"
+ @ParameterizedTest
+ @ValueSource(booleans = Array(true, false))
+ def testRLICreationUsingSQL(isPartitioned: Boolean): Unit = {
Review Comment:
This type of test does not exist in RLI test suite. We are directly calling the index creation SQL commands here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9731] Add index definition for Record level Index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13741:
URL: https://github.com/apache/hudi/pull/13741#issuecomment-3292220171
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7986",
"triggerID" : "f5ff782dfd82d056badb74123ade5c4c127f8421",
"triggerType" : "PUSH"
}, {
"hash" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8048",
"triggerID" : "c2c4ce896ab2c079a41d2660a95649593be84df7",
"triggerType" : "PUSH"
}, {
"hash" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8058",
"triggerID" : "f4b48b8b81c1456289bdfc9df759a8ab4e9b649d",
"triggerType" : "PUSH"
}, {
"hash" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb90a24a3fe161580c467ce14a73e4e7d09f1edc",
"triggerType" : "PUSH"
}, {
"hash" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8060",
"triggerID" : "a42c3c943fa2d31567c56354dc74158968c2b695",
"triggerType" : "PUSH"
}, {
"hash" : "0c91714fcecf6cd348c73d59aec5c5963c690403",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8535",
"triggerID" : "0c91714fcecf6cd348c73d59aec5c5963c690403",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb90a24a3fe161580c467ce14a73e4e7d09f1edc UNKNOWN
* 0c91714fcecf6cd348c73d59aec5c5963c690403 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8535)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org