You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2025/03/05 17:18:59 UTC
[PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
codope opened a new pull request, #12919:
URL: https://github.com/apache/hudi/pull/12919
### Change Logs
Datasource configs already support these options. However, there is no way to provide these options when creating EI using bloom filters via SQL. The sql will fail with validation error. We should support following options through sql as well:
- number of entries
- false positive rate
- bloom filter type
### Impact
index creation parity between datasource and sql
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2710501271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
}, {
"hash" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124",
"triggerID" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"triggerType" : "PUSH"
}, {
"hash" : "40bb2368c49549caaca39485454867d392db5592",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40bb2368c49549caaca39485454867d392db5592",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fadd0bb10f338243630ef81800bcf27c45e9f82 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122)
* 6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124)
* 40bb2368c49549caaca39485454867d392db5592 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2704054407
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3418294c2792119eaf8af3be28a24917725bde23 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2708464016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3418294c2792119eaf8af3be28a24917725bde23 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071)
* 9f453e4fb1d21544ba48d43294e92f7a18741159 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2709836065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f453e4fb1d21544ba48d43294e92f7a18741159 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107)
* 1fadd0bb10f338243630ef81800bcf27c45e9f82 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1987178554
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -328,15 +347,16 @@ public static ExpressionIndexComputationMetadata getExprIndexRecords(List<Pair<S
// Apply expression index and generate the column to index
HoodieExpressionIndex<Column, Column> expressionIndex =
- new HoodieSparkExpressionIndex(indexDefinition.getIndexName(), indexDefinition.getIndexFunction(), indexDefinition.getSourceFields(), indexDefinition.getIndexOptions());
+ new HoodieSparkExpressionIndex(indexDefinition.getIndexName(), indexDefinition.getIndexFunction(), indexDefinition.getIndexType(), indexDefinition.getSourceFields(),
+ indexDefinition.getIndexOptions());
Review Comment:
Refactored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2710786497
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
}, {
"hash" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124",
"triggerID" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"triggerType" : "PUSH"
}, {
"hash" : "40bb2368c49549caaca39485454867d392db5592",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4126",
"triggerID" : "40bb2368c49549caaca39485454867d392db5592",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40bb2368c49549caaca39485454867d392db5592 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4126)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2711565947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
}, {
"hash" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124",
"triggerID" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"triggerType" : "PUSH"
}, {
"hash" : "40bb2368c49549caaca39485454867d392db5592",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4126",
"triggerID" : "40bb2368c49549caaca39485454867d392db5592",
"triggerType" : "PUSH"
}, {
"hash" : "1c9e6a9e3be3bf0d2a1813bd846bc8d4865d73cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c9e6a9e3be3bf0d2a1813bd846bc8d4865d73cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40bb2368c49549caaca39485454867d392db5592 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4126)
* 1c9e6a9e3be3bf0d2a1813bd846bc8d4865d73cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2701765678
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3418294c2792119eaf8af3be28a24917725bde23 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2709832353
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f453e4fb1d21544ba48d43294e92f7a18741159 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107)
* 1fadd0bb10f338243630ef81800bcf27c45e9f82 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2710121028
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fadd0bb10f338243630ef81800bcf27c45e9f82 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1987177787
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -226,6 +230,21 @@ public static ExpressionIndexComputationMetadata getExpressionIndexRecordsUsingB
return new ExpressionIndexComputationMetadata(HoodieJavaRDD.of(bloomFilterRecords.javaRDD()));
}
+ private static void setBloomFilterProps(HoodieWriteConfig metadataWriteConfig, HoodieIndexDefinition indexDefinition) {
+ if (indexDefinition.getIndexOptions().containsKey(HoodieExpressionIndex.FALSE_POSITIVE_RATE)) {
+ metadataWriteConfig.getProps().setProperty(HoodieStorageConfig.BLOOM_FILTER_FPP_VALUE.key(),
+ indexDefinition.getIndexOptions().get(HoodieExpressionIndex.FALSE_POSITIVE_RATE));
+ }
+ if (indexDefinition.getIndexOptions().containsKey(HoodieExpressionIndex.BLOOM_FILTER_NUM_ENTRIES)) {
+ metadataWriteConfig.getProps().setProperty(HoodieStorageConfig.BLOOM_FILTER_NUM_ENTRIES_VALUE.key(),
+ indexDefinition.getIndexOptions().get(HoodieExpressionIndex.BLOOM_FILTER_NUM_ENTRIES));
+ }
+ if (indexDefinition.getIndexOptions().containsKey(HoodieExpressionIndex.BLOOM_FILTER_TYPE)) {
Review Comment:
I missed that. Now it is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1987850649
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -1776,7 +1776,16 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
|""".stripMargin)
// create index using bloom filters on city column with upper() function
- spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper')")
+ spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper', fpp='0.01', filterType='SIMPLE', numEntries='1000')")
+ var metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(s"expr_index_idx_bloom_$tableName"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ assertEquals(2, metaClient.getIndexMetadata.get.getIndexDefinitions.size())
+ val indexDefinition: HoodieIndexDefinition = metaClient.getIndexMetadata.get.getIndexDefinitions.get(s"expr_index_idx_bloom_$tableName")
+ // validate index options
+ assertEquals("0.01", indexDefinition.getIndexOptions.get("fpp"))
+ assertEquals("SIMPLE", indexDefinition.getIndexOptions.get("filterType"))
+ assertEquals("1000", indexDefinition.getIndexOptions.get("numEntries"))
Review Comment:
Sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2710345843
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
}, {
"hash" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124",
"triggerID" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fadd0bb10f338243630ef81800bcf27c45e9f82 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122)
* 6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2710520672
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
}, {
"hash" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124",
"triggerID" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"triggerType" : "PUSH"
}, {
"hash" : "40bb2368c49549caaca39485454867d392db5592",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4126",
"triggerID" : "40bb2368c49549caaca39485454867d392db5592",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124)
* 40bb2368c49549caaca39485454867d392db5592 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4126)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1983802748
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -226,6 +230,21 @@ public static ExpressionIndexComputationMetadata getExpressionIndexRecordsUsingB
return new ExpressionIndexComputationMetadata(HoodieJavaRDD.of(bloomFilterRecords.javaRDD()));
}
+ private static void setBloomFilterProps(HoodieWriteConfig metadataWriteConfig, HoodieIndexDefinition indexDefinition) {
+ if (indexDefinition.getIndexOptions().containsKey(HoodieExpressionIndex.FALSE_POSITIVE_RATE)) {
+ metadataWriteConfig.getProps().setProperty(HoodieStorageConfig.BLOOM_FILTER_FPP_VALUE.key(),
+ indexDefinition.getIndexOptions().get(HoodieExpressionIndex.FALSE_POSITIVE_RATE));
+ }
+ if (indexDefinition.getIndexOptions().containsKey(HoodieExpressionIndex.BLOOM_FILTER_NUM_ENTRIES)) {
+ metadataWriteConfig.getProps().setProperty(HoodieStorageConfig.BLOOM_FILTER_NUM_ENTRIES_VALUE.key(),
+ indexDefinition.getIndexOptions().get(HoodieExpressionIndex.BLOOM_FILTER_NUM_ENTRIES));
+ }
+ if (indexDefinition.getIndexOptions().containsKey(HoodieExpressionIndex.BLOOM_FILTER_TYPE)) {
Review Comment:
Do we consider adding `BLOOM_FILTER_DYNAMIC_MAX_ENTRIES` too?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -204,8 +205,11 @@ public static ExpressionIndexComputationMetadata getExpressionIndexRecordsUsingC
: new ExpressionIndexComputationMetadata(colStatRecords);
}
- public static ExpressionIndexComputationMetadata getExpressionIndexRecordsUsingBloomFilter(Dataset<Row> dataset, String columnToIndex,
- HoodieWriteConfig metadataWriteConfig, String instantTime, String indexName) {
+ public static ExpressionIndexComputationMetadata getExpressionIndexRecordsUsingBloomFilter(Dataset<Row> dataset, String columnToIndex, HoodieWriteConfig metadataWriteConfig, String instantTime,
+ HoodieIndexDefinition indexDefinition) {
+ String indexName = indexDefinition.getIndexName();
+ setBloomFilterProps(metadataWriteConfig, indexDefinition);
Review Comment:
Pass in the `Map<String, String>` options would be better as only the options are needed.
```suggestion
setBloomFilterProps(metadataWriteConfig, indexDefinition.getIndexOptions());
```
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -1776,7 +1776,16 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
|""".stripMargin)
// create index using bloom filters on city column with upper() function
- spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper')")
+ spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper', fpp='0.01', filterType='SIMPLE', numEntries='1000')")
+ var metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(s"expr_index_idx_bloom_$tableName"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ assertEquals(2, metaClient.getIndexMetadata.get.getIndexDefinitions.size())
+ val indexDefinition: HoodieIndexDefinition = metaClient.getIndexMetadata.get.getIndexDefinitions.get(s"expr_index_idx_bloom_$tableName")
+ // validate index options
+ assertEquals("0.01", indexDefinition.getIndexOptions.get("fpp"))
+ assertEquals("SIMPLE", indexDefinition.getIndexOptions.get("filterType"))
+ assertEquals("1000", indexDefinition.getIndexOptions.get("numEntries"))
Review Comment:
looks good. Is it possible to validate that the bloom filters written conform to these params?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -226,6 +230,21 @@ public static ExpressionIndexComputationMetadata getExpressionIndexRecordsUsingB
return new ExpressionIndexComputationMetadata(HoodieJavaRDD.of(bloomFilterRecords.javaRDD()));
}
+ private static void setBloomFilterProps(HoodieWriteConfig metadataWriteConfig, HoodieIndexDefinition indexDefinition) {
+ if (indexDefinition.getIndexOptions().containsKey(HoodieExpressionIndex.FALSE_POSITIVE_RATE)) {
Review Comment:
nit: would be better to have a config key mapping of `Map<String, String>` to store `HoodieExpressionIndex.FALSE_POSITIVE_RATE -> HoodieStorageConfig.BLOOM_FILTER_FPP_VALUE.key()` so we can avoid repeated code on setting each property.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -2161,7 +2169,13 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
df = df.withColumn(HoodieExpressionIndex.HOODIE_EXPRESSION_INDEX_PARTITION, lit("c/d"))
.withColumn(HoodieExpressionIndex.HOODIE_EXPRESSION_INDEX_RELATIVE_FILE_PATH, lit("c/d/123141ab-701b-4ba4-b60b-e6acd9e9103e-0_329-224134-258390_2131313124.parquet"))
.withColumn(HoodieExpressionIndex.HOODIE_EXPRESSION_INDEX_FILE_SIZE, lit(100))
- val bloomFilterRecords = SparkMetadataWriterUtils.getExpressionIndexRecordsUsingBloomFilter(df, "c5", HoodieWriteConfig.newBuilder().withPath("a/b").build(), "", "random")
+ val indexOptions = Map(
+ "fpp"-> "0.01",
+ "numEntries" -> "1000"
+ )
+ val bloomFilterRecords = SparkMetadataWriterUtils.getExpressionIndexRecordsUsingBloomFilter(df, "c5",
+ HoodieWriteConfig.newBuilder().withPath("a/b").build(), "",
+ HoodieIndexDefinition.newBuilder().withIndexName("random").withIndexOptions(JavaConverters.mapAsJavaMapConverter(indexOptions).asJava).build())
Review Comment:
Similar here to see if we can validate the bloom filters conform to the params.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -2161,7 +2169,13 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
df = df.withColumn(HoodieExpressionIndex.HOODIE_EXPRESSION_INDEX_PARTITION, lit("c/d"))
.withColumn(HoodieExpressionIndex.HOODIE_EXPRESSION_INDEX_RELATIVE_FILE_PATH, lit("c/d/123141ab-701b-4ba4-b60b-e6acd9e9103e-0_329-224134-258390_2131313124.parquet"))
.withColumn(HoodieExpressionIndex.HOODIE_EXPRESSION_INDEX_FILE_SIZE, lit(100))
- val bloomFilterRecords = SparkMetadataWriterUtils.getExpressionIndexRecordsUsingBloomFilter(df, "c5", HoodieWriteConfig.newBuilder().withPath("a/b").build(), "", "random")
+ val indexOptions = Map(
+ "fpp"-> "0.01",
+ "numEntries" -> "1000"
Review Comment:
Also add `filterType` or `filter_type`?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -328,15 +347,16 @@ public static ExpressionIndexComputationMetadata getExprIndexRecords(List<Pair<S
// Apply expression index and generate the column to index
HoodieExpressionIndex<Column, Column> expressionIndex =
- new HoodieSparkExpressionIndex(indexDefinition.getIndexName(), indexDefinition.getIndexFunction(), indexDefinition.getSourceFields(), indexDefinition.getIndexOptions());
+ new HoodieSparkExpressionIndex(indexDefinition.getIndexName(), indexDefinition.getIndexFunction(), indexDefinition.getIndexType(), indexDefinition.getSourceFields(),
+ indexDefinition.getIndexOptions());
Review Comment:
Any reason of not passing in the `indexDefinition` as a whole to the constructor?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/index/expression/TestHoodieSparkExpressionIndex.java:
##########
@@ -127,6 +130,7 @@ public void testApplyYearFunctionWithWrongNumberOfArguments() {
HoodieSparkExpressionIndex index = new HoodieSparkExpressionIndex(
"yearIndex",
"year",
+ PARTITION_NAME_COLUMN_STATS,
Review Comment:
Add a new test on `PARTITION_NAME_BLOOM_FILTERS` too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1987854711
##########
hudi-common/src/main/java/org/apache/hudi/index/expression/HoodieExpressionIndex.java:
##########
@@ -47,6 +52,20 @@ public interface HoodieExpressionIndex<S, T> extends Serializable {
String DAYS_OPTION = "days";
String FORMAT_OPTION = "format";
String IDENTITY_TRANSFORM = "identity";
+ // Bloom filter options
+ String BLOOM_FILTER_TYPE = "filtertype";
+ String BLOOM_FILTER_NUM_ENTRIES = "numentries";
+ String FALSE_POSITIVE_RATE = "fpp";
+ String DYNAMIC_BLOOM_MAX_ENTRIES = "maxentries";
Review Comment:
PostgreSQL uses the same pattern of concatenating words without underscore (`_`) for param names so I think this is OK as we follow the same convention.
https://www.postgresql.org/docs/current/sql-createindex.html
```
CREATE UNIQUE INDEX title_idx ON films (title) WITH (fillfactor = 70);
CREATE INDEX gin_idx ON documents_table USING GIN (locations) WITH (fastupdate = off);
```
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -1776,7 +1777,26 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
|""".stripMargin)
// create index using bloom filters on city column with upper() function
- spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper')")
+ spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper', " +
+ s"${HoodieExpressionIndex.FALSE_POSITIVE_RATE}='0.01', ${HoodieExpressionIndex.BLOOM_FILTER_TYPE}='SIMPLE', ${HoodieExpressionIndex.BLOOM_FILTER_NUM_ENTRIES}='1000')")
+ var metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(s"expr_index_idx_bloom_$tableName"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ assertEquals(2, metaClient.getIndexMetadata.get.getIndexDefinitions.size())
+ val indexDefinition: HoodieIndexDefinition = metaClient.getIndexMetadata.get.getIndexDefinitions.get(s"expr_index_idx_bloom_$tableName")
+ // validate index options
+ assertEquals("0.01", indexDefinition.getIndexOptions.get(HoodieExpressionIndex.FALSE_POSITIVE_RATE))
+ assertEquals("SIMPLE", indexDefinition.getIndexOptions.get(HoodieExpressionIndex.BLOOM_FILTER_TYPE))
+ assertEquals("1000", indexDefinition.getIndexOptions.get(HoodieExpressionIndex.BLOOM_FILTER_NUM_ENTRIES))
+
+ // validate index metadata
+ val indexMetadataDf = spark.sql(s"select key, BloomFilterMetadata from hudi_metadata('$tableName') where BloomFilterMetadata is not null")
+ assertEquals(4, indexMetadataDf.count()) // corresponding to 4 riders
Review Comment:
```suggestion
assertEquals(4, indexMetadataDf.count()) // corresponding to 4 files
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #12919:
URL: https://github.com/apache/hudi/pull/12919
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2708483101
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f453e4fb1d21544ba48d43294e92f7a18741159 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2705490013
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3418294c2792119eaf8af3be28a24917725bde23 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2701592805
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3418294c2792119eaf8af3be28a24917725bde23 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1982711030
##########
hudi-common/src/main/java/org/apache/hudi/index/expression/HoodieExpressionIndex.java:
##########
@@ -47,6 +47,10 @@ public interface HoodieExpressionIndex<S, T> extends Serializable {
String DAYS_OPTION = "days";
String FORMAT_OPTION = "format";
String IDENTITY_TRANSFORM = "identity";
+ // Bloom filter options
+ String BLOOM_FILTER_TYPE = "filterType";
+ String BLOOM_FILTER_NUM_ENTRIES = "numEntries";
+ String FALSE_POSITIVE_RATE = "fpp";
Review Comment:
Should we avoid case sensitivity and only use lower case letters in the option names?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -1776,7 +1776,7 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
|""".stripMargin)
// create index using bloom filters on city column with upper() function
- spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper')")
+ spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper', fpp='0.01')")
Review Comment:
Should we test all options? Could you add validation on these params so that they are stored in the index definition and actually take effect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1987179681
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/index/expression/TestHoodieSparkExpressionIndex.java:
##########
@@ -127,6 +130,7 @@ public void testApplyYearFunctionWithWrongNumberOfArguments() {
HoodieSparkExpressionIndex index = new HoodieSparkExpressionIndex(
"yearIndex",
"year",
+ PARTITION_NAME_COLUMN_STATS,
Review Comment:
was already covered in functional test, but now I have added one UT here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1983527764
##########
hudi-common/src/main/java/org/apache/hudi/index/expression/HoodieExpressionIndex.java:
##########
@@ -47,6 +47,10 @@ public interface HoodieExpressionIndex<S, T> extends Serializable {
String DAYS_OPTION = "days";
String FORMAT_OPTION = "format";
String IDENTITY_TRANSFORM = "identity";
+ // Bloom filter options
+ String BLOOM_FILTER_TYPE = "filterType";
+ String BLOOM_FILTER_NUM_ENTRIES = "numEntries";
+ String FALSE_POSITIVE_RATE = "fpp";
Review Comment:
Don't have a string preference. I just thought it's easy to read with camel case. Do we have a convention? I can change if so.
##########
hudi-common/src/main/java/org/apache/hudi/index/expression/HoodieExpressionIndex.java:
##########
@@ -47,6 +47,10 @@ public interface HoodieExpressionIndex<S, T> extends Serializable {
String DAYS_OPTION = "days";
String FORMAT_OPTION = "format";
String IDENTITY_TRANSFORM = "identity";
+ // Bloom filter options
+ String BLOOM_FILTER_TYPE = "filterType";
+ String BLOOM_FILTER_NUM_ENTRIES = "numEntries";
+ String FALSE_POSITIVE_RATE = "fpp";
Review Comment:
Don't have a strong preference. I just thought it's easy to read with camel case. Do we have a convention? I can change if so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1983523264
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -1776,7 +1776,7 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
|""".stripMargin)
// create index using bloom filters on city column with upper() function
- spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper')")
+ spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper', fpp='0.01')")
Review Comment:
Sure. Now i added all options and also validating that in index metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1983752418
##########
hudi-common/src/main/java/org/apache/hudi/index/expression/HoodieExpressionIndex.java:
##########
@@ -47,6 +47,10 @@ public interface HoodieExpressionIndex<S, T> extends Serializable {
String DAYS_OPTION = "days";
String FORMAT_OPTION = "format";
String IDENTITY_TRANSFORM = "identity";
+ // Bloom filter options
+ String BLOOM_FILTER_TYPE = "filterType";
+ String BLOOM_FILTER_NUM_ENTRIES = "numEntries";
+ String FALSE_POSITIVE_RATE = "fpp";
Review Comment:
Usually SQL statements are case insensitive so I think it's better to use all lowercase letters and underscore for word separation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2701596729
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3418294c2792119eaf8af3be28a24917725bde23 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2708463513
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3418294c2792119eaf8af3be28a24917725bde23 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071)
* 9f453e4fb1d21544ba48d43294e92f7a18741159 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2710342641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
}, {
"hash" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fadd0bb10f338243630ef81800bcf27c45e9f82 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122)
* 6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12919:
URL: https://github.com/apache/hudi/pull/12919#discussion_r1987184769
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/index/TestExpressionIndex.scala:
##########
@@ -1776,7 +1776,16 @@ class TestExpressionIndex extends HoodieSparkSqlTestBase {
|""".stripMargin)
// create index using bloom filters on city column with upper() function
- spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper')")
+ spark.sql(s"create index idx_bloom_$tableName on $tableName using bloom_filters(city) options(expr='upper', fpp='0.01', filterType='SIMPLE', numEntries='1000')")
+ var metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(s"expr_index_idx_bloom_$tableName"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ assertEquals(2, metaClient.getIndexMetadata.get.getIndexDefinitions.size())
+ val indexDefinition: HoodieIndexDefinition = metaClient.getIndexMetadata.get.getIndexDefinitions.get(s"expr_index_idx_bloom_$tableName")
+ // validate index options
+ assertEquals("0.01", indexDefinition.getIndexOptions.get("fpp"))
+ assertEquals("SIMPLE", indexDefinition.getIndexOptions.get("filterType"))
+ assertEquals("1000", indexDefinition.getIndexOptions.get("numEntries"))
Review Comment:
among these options, we are only saving the bloom filter type in bloom filter metadata.. Added a validation in both the tests on the type (one test validates SIMPLE and the other DYNAMIC_V0). Since the type is getting propagated, every other option should also get propagated properly. Also, there is validaiton on the index definition that we save.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8768] Support bloom filter options when creating expr index using bloom filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12919:
URL: https://github.com/apache/hudi/pull/12919#issuecomment-2710513167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3418294c2792119eaf8af3be28a24917725bde23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4071",
"triggerID" : "3418294c2792119eaf8af3be28a24917725bde23",
"triggerType" : "PUSH"
}, {
"hash" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4107",
"triggerID" : "9f453e4fb1d21544ba48d43294e92f7a18741159",
"triggerType" : "PUSH"
}, {
"hash" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4122",
"triggerID" : "1fadd0bb10f338243630ef81800bcf27c45e9f82",
"triggerType" : "PUSH"
}, {
"hash" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124",
"triggerID" : "6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b",
"triggerType" : "PUSH"
}, {
"hash" : "40bb2368c49549caaca39485454867d392db5592",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40bb2368c49549caaca39485454867d392db5592",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bfe27d59a1b96a6d8d5ef6771f4ad8155aab45b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4124)
* 40bb2368c49549caaca39485454867d392db5592 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org