You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2023/10/16 16:30:06 UTC
[PR] [HUDI-5210][WIP] Implement functional indexes [hudi]
codope opened a new pull request, #9872:
URL: https://github.com/apache/hudi/pull/9872
### Change Logs
Support functional indexes.
### Impact
Users can now create indexes on not jsut columns but also functions based on columns, e.g. `DATE_FORMAT(ts, '%Y-%m-%d')`
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788430760
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579)
* eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788662479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579)
* eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602)
* 120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379931555
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -422,6 +445,9 @@ private boolean initializeFromFilesystem(String initializationTime, List<Metadat
case RECORD_INDEX:
fileGroupCountAndRecordsPair = initializeRecordIndexPartition();
break;
+ case FUNCTIONAL_INDEX:
+ fileGroupCountAndRecordsPair = initializeFunctionalIndexPartition();
Review Comment:
Right now, you can jsut create one partition with a create index command (or using async indexer). So, the code assumes that user is not creating multiple functional indexes in the same job but we can extend the functionality in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1782202405
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* babdb27477a99fd7f70d67d2ece7ca484ccf6f8b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1789265768
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788650867
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579)
* eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602)
* 120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1785011007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ce109f21907405da53feb132faaa567839662bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379919844
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
+ public static final String INDEX_DEFINITION_FILE_BACKUP = "index.properties.backup";
+ public static final ConfigProperty<String> INDEX_NAME = ConfigProperty
+ .key("hoodie.functional.index.name")
+ .noDefaultValue()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Name of the functional index. This is also used for the partition name in the metadata table.");
+
+ public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
+ .key("hoodie.functional.index.type")
+ .noDefaultValue(MetadataPartitionType.COLUMN_STATS.name())
+ .withValidValues(
+ MetadataPartitionType.FILES.name(),
+ MetadataPartitionType.COLUMN_STATS.name(),
+ MetadataPartitionType.BLOOM_FILTERS.name(),
+ MetadataPartitionType.RECORD_INDEX.name(),
+ SecondaryIndexType.LUCENE.name()
Review Comment:
no not yet.. removed and keeping only the ones supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379928885
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -205,17 +216,27 @@ private void initMetadataReader() {
*/
private void enablePartitions() {
final HoodieMetadataConfig metadataConfig = dataWriteConfig.getMetadataConfig();
- if (dataWriteConfig.isMetadataTableEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.FILES)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.FILES);
+ if (dataWriteConfig.isMetadataTableEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(FILES)) {
+ this.enabledPartitionTypes.add(FILES);
}
- if (metadataConfig.isBloomFilterIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.BLOOM_FILTERS)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.BLOOM_FILTERS);
+ if (metadataConfig.isBloomFilterIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(BLOOM_FILTERS)) {
+ this.enabledPartitionTypes.add(BLOOM_FILTERS);
}
- if (metadataConfig.isColumnStatsIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.COLUMN_STATS)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.COLUMN_STATS);
+ if (metadataConfig.isColumnStatsIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(COLUMN_STATS)) {
+ this.enabledPartitionTypes.add(COLUMN_STATS);
}
- if (dataWriteConfig.isRecordIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.RECORD_INDEX)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.RECORD_INDEX);
+ if (dataWriteConfig.isRecordIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(RECORD_INDEX)) {
+ this.enabledPartitionTypes.add(RECORD_INDEX);
+ }
+ Option<HoodieFunctionalIndexConfig> functionalIndexConfig = dataMetaClient.getFunctionalIndexConfig();
+ if (functionalIndexConfig.isPresent()) {
+ this.enabledPartitionTypes.add(FUNCTIONAL_INDEX);
Review Comment:
create index will add a new partition and also the table config for available metadata partitions will be updated. That way, readers can use the index as well as subsequent writes can update the index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1784739280
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* babdb27477a99fd7f70d67d2ece7ca484ccf6f8b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524)
* 455fcaddf8f14051da29eacf37ce7e34c6208886 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555)
* 8ce109f21907405da53feb132faaa567839662bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #9872:
URL: https://github.com/apache/hudi/pull/9872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1782282027
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* babdb27477a99fd7f70d67d2ece7ca484ccf6f8b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1784725418
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* babdb27477a99fd7f70d67d2ece7ca484ccf6f8b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524)
* 455fcaddf8f14051da29eacf37ce7e34c6208886 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555)
* 8ce109f21907405da53feb132faaa567839662bf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1380906029
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
+ public static final String INDEX_DEFINITION_FILE_BACKUP = "index.properties.backup";
+ public static final ConfigProperty<String> INDEX_NAME = ConfigProperty
Review Comment:
Got it.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
Review Comment:
No need to make the decision in this PR. We should follow up on the design around where the index definitions are stored.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -422,6 +445,9 @@ private boolean initializeFromFilesystem(String initializationTime, List<Metadat
case RECORD_INDEX:
fileGroupCountAndRecordsPair = initializeRecordIndexPartition();
break;
+ case FUNCTIONAL_INDEX:
+ fileGroupCountAndRecordsPair = initializeFunctionalIndexPartition();
Review Comment:
I see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379941270
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/functional/HoodieSparkFunctionalIndex.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.index.functional;
+
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.functions;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+
+public class HoodieSparkFunctionalIndex implements HoodieFunctionalIndex<Column, Column>, Serializable {
+
+ /**
+ * Custom interface to support Spark functions
+ */
+ @FunctionalInterface
+ interface SparkFunction {
+ Column apply(List<Column> columns, Map<String, String> options);
+ }
+
+ /**
+ * Map of Spark functions to their implementations.
+ * NOTE: This is not an exhaustive list of spark-sql functions. Only the common date/timestamp and string functions have been added.
+ * Add more functions as needed. However, keep the key should match the exact spark-sql function name in lowercase.
+ */
+ private static final Map<String, SparkFunction> SPARK_FUNCTION_MAP = CollectionUtils.createImmutableMap(
Review Comment:
Didn't consider. Will add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379919453
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
+ public static final String INDEX_DEFINITION_FILE_BACKUP = "index.properties.backup";
+ public static final ConfigProperty<String> INDEX_NAME = ConfigProperty
+ .key("hoodie.functional.index.name")
+ .noDefaultValue()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Name of the functional index. This is also used for the partition name in the metadata table.");
+
+ public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
+ .key("hoodie.functional.index.type")
+ .noDefaultValue(MetadataPartitionType.COLUMN_STATS.name())
Review Comment:
corrected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379931555
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -422,6 +445,9 @@ private boolean initializeFromFilesystem(String initializationTime, List<Metadat
case RECORD_INDEX:
fileGroupCountAndRecordsPair = initializeRecordIndexPartition();
break;
+ case FUNCTIONAL_INDEX:
+ fileGroupCountAndRecordsPair = initializeFunctionalIndexPartition();
Review Comment:
Right now, you can jsut create one partition with a create index command (or using async indexer). So, the code assumes that user is not creating multiple functional indexes in the same job but we can extend the functionality in a followup. [HUDI-7026](https://issues.apache.org/jira/browse/HUDI-7026) to track.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1791439149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
}, {
"hash" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640",
"triggerID" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"triggerType" : "PUSH"
}, {
"hash" : "0d2dace457162b24edaabe2c83b8d6d0c310050a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20643",
"triggerID" : "0d2dace457162b24edaabe2c83b8d6d0c310050a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2eced526259327f5abfb8ac92d8b37b7a4b12c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640)
* 0d2dace457162b24edaabe2c83b8d6d0c310050a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20643)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1377086649
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
+ public static final String INDEX_DEFINITION_FILE_BACKUP = "index.properties.backup";
+ public static final ConfigProperty<String> INDEX_NAME = ConfigProperty
+ .key("hoodie.functional.index.name")
+ .noDefaultValue()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Name of the functional index. This is also used for the partition name in the metadata table.");
+
+ public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
+ .key("hoodie.functional.index.type")
+ .noDefaultValue(MetadataPartitionType.COLUMN_STATS.name())
+ .withValidValues(
+ MetadataPartitionType.FILES.name(),
+ MetadataPartitionType.COLUMN_STATS.name(),
+ MetadataPartitionType.BLOOM_FILTERS.name(),
+ MetadataPartitionType.RECORD_INDEX.name(),
+ SecondaryIndexType.LUCENE.name()
Review Comment:
Are all of these supported by this PR?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
+ public static final String INDEX_DEFINITION_FILE_BACKUP = "index.properties.backup";
+ public static final ConfigProperty<String> INDEX_NAME = ConfigProperty
Review Comment:
Are multiple functional indexes supported by the new configs? e.g., if I'd like to create two indexes based on `month(ts)` with `FILES` and `day(ts)` with `COLUMN_STATS`.
##########
hudi-common/src/main/java/org/apache/hudi/index/secondary/HoodieSecondaryIndex.java:
##########
@@ -17,7 +17,7 @@
* under the License.
*/
-package org.apache.hudi.secondary.index;
+package org.apache.hudi.index.secondary;
Review Comment:
Is this class a public API that a user can extend? If not, changing the package name is OK.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -316,6 +316,20 @@ public final class HoodieMetadataConfig extends HoodieConfig {
.withDocumentation("Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. "
+ "Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.");
+ public static final ConfigProperty<Integer> FUNCTIONAL_INDEX_FILE_GROUP_COUNT = ConfigProperty
+ .key(METADATA_PREFIX + ".index.functional.file.group.count")
+ .defaultValue(2)
+ .markAdvanced()
+ .sinceVersion("0.11.0")
Review Comment:
nit: `1.0.0`
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -28,7 +28,8 @@ public enum MetadataPartitionType {
FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-"),
COLUMN_STATS(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS, "col-stats-"),
BLOOM_FILTERS(HoodieTableMetadataUtil.PARTITION_NAME_BLOOM_FILTERS, "bloom-filters-"),
- RECORD_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX, "record-index-");
+ RECORD_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX, "record-index-"),
+ FUNCTIONAL_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_FUNCTIONAL_INDEX_PREFIX, "func_index_");
Review Comment:
Use dashes `-` to be consistent with other partition names?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
+ public static final String INDEX_DEFINITION_FILE_BACKUP = "index.properties.backup";
+ public static final ConfigProperty<String> INDEX_NAME = ConfigProperty
+ .key("hoodie.functional.index.name")
+ .noDefaultValue()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Name of the functional index. This is also used for the partition name in the metadata table.");
+
+ public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
+ .key("hoodie.functional.index.type")
+ .noDefaultValue(MetadataPartitionType.COLUMN_STATS.name())
Review Comment:
Should this be `.defaultValue(MetadataPartitionType.COLUMN_STATS.name())`?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
Review Comment:
Should the functional index metadata be stored along with the table properties, i.e., `hoodie.properties`? Another file is another RPC call on the cloud storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1785114404
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ce109f21907405da53feb132faaa567839662bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557)
* 65d4f829b812cc7c76c0efb5cc03658ce2c88f29 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1790873344
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
}, {
"hash" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640",
"triggerID" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2eced526259327f5abfb8ac92d8b37b7a4b12c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379930246
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -498,6 +524,40 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeBloomFiltersPartition(
return Pair.of(fileGroupCount, records);
}
+ protected abstract HoodieData<HoodieRecord> getFunctionalIndexRecords(List<Pair<String, FileSlice>> partitionFileSlicePairs,
+ HoodieFunctionalIndexDefinition indexDefinition,
+ HoodieTableMetaClient metaClient,
+ int parallelism, Schema readerSchema,
+ SerializableConfiguration hadoopConf);
+
+ private Pair<Integer, HoodieData<HoodieRecord>> initializeFunctionalIndexPartition() throws Exception {
+ HoodieMetadataFileSystemView fsView = new HoodieMetadataFileSystemView(dataMetaClient, dataMetaClient.getActiveTimeline(), metadata);
+ HoodieFunctionalIndexDefinition indexDefinition = getFunctionalIndexDefinition("");
Review Comment:
this is fixed. it's an old left over code which has been refactored now based on `HoodieFunctionalIndexDefinition`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788313054
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 65d4f829b812cc7c76c0efb5cc03658ce2c88f29 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562)
* 475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788299564
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 65d4f829b812cc7c76c0efb5cc03658ce2c88f29 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562)
* 475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1782208248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* babdb27477a99fd7f70d67d2ece7ca484ccf6f8b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1784646472
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* babdb27477a99fd7f70d67d2ece7ca484ccf6f8b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524)
* 455fcaddf8f14051da29eacf37ce7e34c6208886 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1785388810
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 65d4f829b812cc7c76c0efb5cc03658ce2c88f29 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379931834
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/functional/BaseHoodieFunctionalIndexClient.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.table.action.index.functional;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+public abstract class BaseHoodieFunctionalIndexClient {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseHoodieFunctionalIndexClient.class);
+
+ public BaseHoodieFunctionalIndexClient() {
+ }
+
+ public void register(HoodieTableMetaClient metaClient, String indexName, String indexType, Map<String, Map<String, String>> columns, Map<String, String> options) {
Review Comment:
Added java docs and test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1791705100
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
}, {
"hash" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640",
"triggerID" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"triggerType" : "PUSH"
}, {
"hash" : "0d2dace457162b24edaabe2c83b8d6d0c310050a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20643",
"triggerID" : "0d2dace457162b24edaabe2c83b8d6d0c310050a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d2dace457162b24edaabe2c83b8d6d0c310050a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20643)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379942448
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigGroups.java:
##########
@@ -40,7 +40,8 @@ public enum Names {
RECORD_PAYLOAD("Record Payload Config"),
KAFKA_CONNECT("Kafka Connect Configs"),
AWS("Amazon Web Services Configs"),
- HUDI_STREAMER("Hudi Streamer Configs");
+ HUDI_STREAMER("Hudi Streamer Configs"),
+ INDEXING("Indexing Configs");
Review Comment:
Yeah I'm slightly biased towards INDEXING/INDEXER instead of INDEX. Let me take that it in a seprate PR. It will touch more files. [HUDI-7025](https://issues.apache.org/jira/browse/HUDI-7025) to track
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1790559044
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
}, {
"hash" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74cc5ad3650edbb7aee3e98811ee680398bc0dd0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627)
* d2eced526259327f5abfb8ac92d8b37b7a4b12c2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1790570781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
}, {
"hash" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640",
"triggerID" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74cc5ad3650edbb7aee3e98811ee680398bc0dd0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627)
* d2eced526259327f5abfb8ac92d8b37b7a4b12c2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788391577
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788422635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579)
* eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1788802555
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602)
* 120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1784825658
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 455fcaddf8f14051da29eacf37ce7e34c6208886 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555)
* 8ce109f21907405da53feb132faaa567839662bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1785036617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ce109f21907405da53feb132faaa567839662bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557)
* 65d4f829b812cc7c76c0efb5cc03658ce2c88f29 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379924242
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
Review Comment:
Agree with you but there can be various secondary indexes and we wanted to keep the index definitions decoupled from the table location - https://github.com/apache/hudi/pull/7235#discussion_r1322226783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1791428572
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
}, {
"hash" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640",
"triggerID" : "d2eced526259327f5abfb8ac92d8b37b7a4b12c2",
"triggerType" : "PUSH"
}, {
"hash" : "0d2dace457162b24edaabe2c83b8d6d0c310050a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d2dace457162b24edaabe2c83b8d6d0c310050a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2eced526259327f5abfb8ac92d8b37b7a4b12c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20640)
* 0d2dace457162b24edaabe2c83b8d6d0c310050a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379974822
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/functional/HoodieSparkFunctionalIndex.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.index.functional;
+
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.functions;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+
+public class HoodieSparkFunctionalIndex implements HoodieFunctionalIndex<Column, Column>, Serializable {
+
+ /**
+ * Custom interface to support Spark functions
+ */
+ @FunctionalInterface
+ interface SparkFunction {
+ Column apply(List<Column> columns, Map<String, String> options);
+ }
+
+ /**
+ * Map of Spark functions to their implementations.
+ * NOTE: This is not an exhaustive list of spark-sql functions. Only the common date/timestamp and string functions have been added.
+ * Add more functions as needed. However, keep the key should match the exact spark-sql function name in lowercase.
+ */
+ private static final Map<String, SparkFunction> SPARK_FUNCTION_MAP = CollectionUtils.createImmutableMap(
Review Comment:
have added it now in `HoodieSparkFunctionalIndex`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1789919716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611)
* 74cc5ad3650edbb7aee3e98811ee680398bc0dd0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1790006589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74cc5ad3650edbb7aee3e98811ee680398bc0dd0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1784658210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* babdb27477a99fd7f70d67d2ece7ca484ccf6f8b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524)
* 455fcaddf8f14051da29eacf37ce7e34c6208886 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1377108821
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -205,17 +216,27 @@ private void initMetadataReader() {
*/
private void enablePartitions() {
final HoodieMetadataConfig metadataConfig = dataWriteConfig.getMetadataConfig();
- if (dataWriteConfig.isMetadataTableEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.FILES)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.FILES);
+ if (dataWriteConfig.isMetadataTableEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(FILES)) {
+ this.enabledPartitionTypes.add(FILES);
}
- if (metadataConfig.isBloomFilterIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.BLOOM_FILTERS)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.BLOOM_FILTERS);
+ if (metadataConfig.isBloomFilterIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(BLOOM_FILTERS)) {
+ this.enabledPartitionTypes.add(BLOOM_FILTERS);
}
- if (metadataConfig.isColumnStatsIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.COLUMN_STATS)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.COLUMN_STATS);
+ if (metadataConfig.isColumnStatsIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(COLUMN_STATS)) {
+ this.enabledPartitionTypes.add(COLUMN_STATS);
}
- if (dataWriteConfig.isRecordIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.RECORD_INDEX)) {
- this.enabledPartitionTypes.add(MetadataPartitionType.RECORD_INDEX);
+ if (dataWriteConfig.isRecordIndexEnabled() || dataMetaClient.getTableConfig().isMetadataPartitionAvailable(RECORD_INDEX)) {
+ this.enabledPartitionTypes.add(RECORD_INDEX);
+ }
+ Option<HoodieFunctionalIndexConfig> functionalIndexConfig = dataMetaClient.getFunctionalIndexConfig();
+ if (functionalIndexConfig.isPresent()) {
+ this.enabledPartitionTypes.add(FUNCTIONAL_INDEX);
Review Comment:
I'm trying to understand how `FUNCTIONAL_INDEX` is enabled. Based on the code, `CREATE INDEX` can add a new functional index partition. What about the write configs? Are users allowed to create a functional index through write configs?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigGroups.java:
##########
@@ -40,7 +40,8 @@ public enum Names {
RECORD_PAYLOAD("Record Payload Config"),
KAFKA_CONNECT("Kafka Connect Configs"),
AWS("Amazon Web Services Configs"),
- HUDI_STREAMER("Hudi Streamer Configs");
+ HUDI_STREAMER("Hudi Streamer Configs"),
+ INDEXING("Indexing Configs");
Review Comment:
There is `INDEX` sub-group name below. Should that be consolidated?
```
INDEX(
"Index Configs",
"Configurations that control indexing behavior, "
+ "which tags incoming records as either inserts or updates to older records."),
```
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -498,6 +524,40 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeBloomFiltersPartition(
return Pair.of(fileGroupCount, records);
}
+ protected abstract HoodieData<HoodieRecord> getFunctionalIndexRecords(List<Pair<String, FileSlice>> partitionFileSlicePairs,
+ HoodieFunctionalIndexDefinition indexDefinition,
+ HoodieTableMetaClient metaClient,
+ int parallelism, Schema readerSchema,
+ SerializableConfiguration hadoopConf);
+
+ private Pair<Integer, HoodieData<HoodieRecord>> initializeFunctionalIndexPartition() throws Exception {
+ HoodieMetadataFileSystemView fsView = new HoodieMetadataFileSystemView(dataMetaClient, dataMetaClient.getActiveTimeline(), metadata);
+ HoodieFunctionalIndexDefinition indexDefinition = getFunctionalIndexDefinition("");
Review Comment:
Why is the index name empty?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/functional/HoodieSparkFunctionalIndex.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.index.functional;
+
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.functions;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+
+public class HoodieSparkFunctionalIndex implements HoodieFunctionalIndex<Column, Column>, Serializable {
+
+ /**
+ * Custom interface to support Spark functions
+ */
+ @FunctionalInterface
+ interface SparkFunction {
+ Column apply(List<Column> columns, Map<String, String> options);
+ }
+
+ /**
+ * Map of Spark functions to their implementations.
+ * NOTE: This is not an exhaustive list of spark-sql functions. Only the common date/timestamp and string functions have been added.
+ * Add more functions as needed. However, keep the key should match the exact spark-sql function name in lowercase.
+ */
+ private static final Map<String, SparkFunction> SPARK_FUNCTION_MAP = CollectionUtils.createImmutableMap(
Review Comment:
Do we consider identity function, i.e., the column itself?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -422,6 +445,9 @@ private boolean initializeFromFilesystem(String initializationTime, List<Metadat
case RECORD_INDEX:
fileGroupCountAndRecordsPair = initializeRecordIndexPartition();
break;
+ case FUNCTIONAL_INDEX:
+ fileGroupCountAndRecordsPair = initializeFunctionalIndexPartition();
Review Comment:
Does each individual function have its only MDT partition, e.g., `metadata/functional_index_date`, `metadata/functional_index_month`? Should this consider multiple partitions?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/functional/BaseHoodieFunctionalIndexClient.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.table.action.index.functional;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+public abstract class BaseHoodieFunctionalIndexClient {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseHoodieFunctionalIndexClient.class);
+
+ public BaseHoodieFunctionalIndexClient() {
+ }
+
+ public void register(HoodieTableMetaClient metaClient, String indexName, String indexType, Map<String, Map<String, String>> columns, Map<String, String> options) {
Review Comment:
An example in the javadocs of how an index definition is retrieved and serialized into the index metadata under `.hoodie` would help here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9872:
URL: https://github.com/apache/hudi/pull/9872#issuecomment-1789926650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20524",
"triggerID" : "babdb27477a99fd7f70d67d2ece7ca484ccf6f8b",
"triggerType" : "PUSH"
}, {
"hash" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20555",
"triggerID" : "455fcaddf8f14051da29eacf37ce7e34c6208886",
"triggerType" : "PUSH"
}, {
"hash" : "8ce109f21907405da53feb132faaa567839662bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20557",
"triggerID" : "8ce109f21907405da53feb132faaa567839662bf",
"triggerType" : "PUSH"
}, {
"hash" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20562",
"triggerID" : "65d4f829b812cc7c76c0efb5cc03658ce2c88f29",
"triggerType" : "PUSH"
}, {
"hash" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20579",
"triggerID" : "475e0edf5b41a2ec0aec5b23b9852f814e0b4aa4",
"triggerType" : "PUSH"
}, {
"hash" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20602",
"triggerID" : "eb0a46891bcf42c28c66b2eb7e2429cfcdb8b63d",
"triggerType" : "PUSH"
}, {
"hash" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611",
"triggerID" : "120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0",
"triggerType" : "PUSH"
}, {
"hash" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627",
"triggerID" : "74cc5ad3650edbb7aee3e98811ee680398bc0dd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 120afe45c4eb6e5ca6fa51dc2614c90f50c3dec0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20611)
* 74cc5ad3650edbb7aee3e98811ee680398bc0dd0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20627)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379942448
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigGroups.java:
##########
@@ -40,7 +40,8 @@ public enum Names {
RECORD_PAYLOAD("Record Payload Config"),
KAFKA_CONNECT("Kafka Connect Configs"),
AWS("Amazon Web Services Configs"),
- HUDI_STREAMER("Hudi Streamer Configs");
+ HUDI_STREAMER("Hudi Streamer Configs"),
+ INDEXING("Indexing Configs");
Review Comment:
Yeah I'm slightly biased towards INDEXING/INDEXER instead of INDEX. Let me take that it in a seprate PR. It will touch more files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379926300
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieFunctionalIndexConfig.java:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.util.BinaryUtil;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.index.secondary.SecondaryIndexType;
+import org.apache.hudi.metadata.MetadataPartitionType;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.time.Instant;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import static org.apache.hudi.common.util.ConfigUtils.fetchConfigs;
+import static org.apache.hudi.common.util.ConfigUtils.recoverIfNeeded;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+
+@Immutable
+@ConfigClassProperty(name = "Common Index Configs",
+ groupName = ConfigGroups.Names.INDEXING,
+ subGroupName = ConfigGroups.SubGroupNames.FUNCTIONAL_INDEX,
+ areCommonConfigs = true,
+ description = "")
+public class HoodieFunctionalIndexConfig extends HoodieConfig {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieFunctionalIndexConfig.class);
+
+ public static final String INDEX_DEFINITION_FILE = "index.properties";
+ public static final String INDEX_DEFINITION_FILE_BACKUP = "index.properties.backup";
+ public static final ConfigProperty<String> INDEX_NAME = ConfigProperty
Review Comment:
yes you can, but you need to run two separate index commands (essentially trigger two different jobs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1379927596
##########
hudi-common/src/main/java/org/apache/hudi/index/secondary/HoodieSecondaryIndex.java:
##########
@@ -17,7 +17,7 @@
* under the License.
*/
-package org.apache.hudi.secondary.index;
+package org.apache.hudi.index.secondary;
Review Comment:
Not public yet. There is no concrete impl in any released version or master. There is a PR in review - #6677
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -28,7 +28,8 @@ public enum MetadataPartitionType {
FILES(HoodieTableMetadataUtil.PARTITION_NAME_FILES, "files-"),
COLUMN_STATS(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS, "col-stats-"),
BLOOM_FILTERS(HoodieTableMetadataUtil.PARTITION_NAME_BLOOM_FILTERS, "bloom-filters-"),
- RECORD_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX, "record-index-");
+ RECORD_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX, "record-index-"),
+ FUNCTIONAL_INDEX(HoodieTableMetadataUtil.PARTITION_NAME_FUNCTIONAL_INDEX_PREFIX, "func_index_");
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5210] Implement functional indexes [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9872:
URL: https://github.com/apache/hudi/pull/9872#discussion_r1380966712
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigGroups.java:
##########
@@ -40,7 +40,8 @@ public enum Names {
RECORD_PAYLOAD("Record Payload Config"),
KAFKA_CONNECT("Kafka Connect Configs"),
AWS("Amazon Web Services Configs"),
- HUDI_STREAMER("Hudi Streamer Configs");
+ HUDI_STREAMER("Hudi Streamer Configs"),
+ INDEXING("Indexing Configs");
Review Comment:
In that case, let's remove the subgroup of `INDEX` or rename it sth different in the follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org