You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/23 11:59:39 UTC
[GitHub] [pinot] saurabhd336 opened a new pull request, #8961: (WIP) Allow bloom filter for no dictionary SV columns
saurabhd336 opened a new pull request, #8961:
URL: https://github.com/apache/pinot/pull/8961
Instructions:
1. The PR has to be tagged with at least one of the following labels (*):
1. `feature`
2. `bugfix`
3. `performance`
4. `ui`
5. `backward-incompat`
6. `release-notes` (**)
2. Remove these instructions before publishing the PR.
(*) Other labels to consider:
- `testing`
- `dependencies`
- `docker`
- `kubernetes`
- `observability`
- `security`
- `code-style`
- `extension-point`
- `refactor`
- `cleanup`
(**) Use `release-notes` label for scenarios like:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8961:
URL: https://github.com/apache/pinot/pull/8961#issuecomment-1165320531
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8961?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8961](https://codecov.io/gh/apache/pinot/pull/8961?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b62eb58) into [master](https://codecov.io/gh/apache/pinot/commit/47c637c38c5f9dfc0c79efd95fb761a4a692cfde?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47c637c) will **decrease** coverage by `44.61%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #8961 +/- ##
=============================================
- Coverage 69.71% 25.09% -44.62%
+ Complexity 4710 47 -4663
=============================================
Files 1817 1805 -12
Lines 94789 94522 -267
Branches 14178 14163 -15
=============================================
- Hits 66083 23724 -42359
- Misses 24065 68493 +44428
+ Partials 4641 2305 -2336
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `25.09% <0.00%> (+0.16%)` | :arrow_up: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8961?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ocal/indexsegment/mutable/IntermediateSegment.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9JbnRlcm1lZGlhdGVTZWdtZW50LmphdmE=) | `0.00% <0.00%> (-76.52%)` | :arrow_down: |
| [...local/indexsegment/mutable/MutableSegmentImpl.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9NdXRhYmxlU2VnbWVudEltcGwuamF2YQ==) | `0.00% <0.00%> (-58.64%)` | :arrow_down: |
| [...verter/stats/MutableNoDictionaryColStatistics.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9jb252ZXJ0ZXIvc3RhdHMvTXV0YWJsZU5vRGljdGlvbmFyeUNvbFN0YXRpc3RpY3MuamF2YQ==) | `0.00% <0.00%> (-42.86%)` | :arrow_down: |
| [...gment/index/column/IntermediateIndexContainer.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2NvbHVtbi9JbnRlcm1lZGlhdGVJbmRleENvbnRhaW5lci5qYXZh) | `0.00% <0.00%> (-81.58%)` | :arrow_down: |
| [...ocal/segment/index/datasource/EmptyDataSource.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2RhdGFzb3VyY2UvRW1wdHlEYXRhU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../segment/index/datasource/ImmutableDataSource.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2RhdGFzb3VyY2UvSW1tdXRhYmxlRGF0YVNvdXJjZS5qYXZh) | `0.00% <0.00%> (-92.86%)` | :arrow_down: |
| [...al/segment/index/datasource/MutableDataSource.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2RhdGFzb3VyY2UvTXV0YWJsZURhdGFTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (-87.50%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-80.83%)` | :arrow_down: |
| [...nt/local/startree/v2/store/StarTreeDataSource.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9zdG9yZS9TdGFyVHJlZURhdGFTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (-40.00%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/TableConfigUtils.java](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9UYWJsZUNvbmZpZ1V0aWxzLmphdmE=) | `0.00% <ø> (-67.92%)` | :arrow_down: |
| ... and [1339 more](https://codecov.io/gh/apache/pinot/pull/8961/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8961?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8961?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [47c637c...b62eb58](https://codecov.io/gh/apache/pinot/pull/8961?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8961:
URL: https://github.com/apache/pinot/pull/8961#discussion_r919457077
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -136,17 +138,132 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
bloomFilterConfig);
- try (BloomFilterCreator bloomFilterCreator = indexCreatorProvider.newBloomFilterCreator(
- IndexCreationContext.builder().withIndexDir(indexDir).withColumnMetadata(columnMetadata)
- .build().forBloomFilter(bloomFilterConfig));
- Dictionary dictionary = getDictionaryReader(columnMetadata, segmentWriter)) {
- int length = dictionary.length();
- for (int i = 0; i < length; i++) {
- bloomFilterCreator.add(dictionary.getStringValue(i));
+
+ if (columnMetadata.hasDictionary()) {
Review Comment:
(nit) suggest separating this into different methods for readability. Something like ....
```
createBloomFilterOnDictColumn
createBloomFilterOnNoDictSVColumn
createBloomFilterOnNoDictMVColumn
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8961:
URL: https://github.com/apache/pinot/pull/8961#discussion_r919477142
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/datasource/ImmutableDataSource.java:
##########
@@ -104,6 +104,12 @@ public Comparable getMaxValue() {
return _maxValue;
}
+ @Nullable
+ @Override
+ public Set<Comparable> getUniqueValueSet() {
Review Comment:
May be we can see if the `DataSource` interface change can be avoided esp if this is applicable only to mutable ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8961:
URL: https://github.com/apache/pinot/pull/8961#discussion_r919477142
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/datasource/ImmutableDataSource.java:
##########
@@ -104,6 +104,12 @@ public Comparable getMaxValue() {
return _maxValue;
}
+ @Nullable
+ @Override
+ public Set<Comparable> getUniqueValueSet() {
Review Comment:
Can we avoid changing the `DataSource` interface if this is applicable only to mutable ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8961:
URL: https://github.com/apache/pinot/pull/8961#discussion_r919477142
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/datasource/ImmutableDataSource.java:
##########
@@ -104,6 +104,12 @@ public Comparable getMaxValue() {
return _maxValue;
}
+ @Nullable
+ @Override
+ public Set<Comparable> getUniqueValueSet() {
Review Comment:
Don't we need to change the constructor as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on PR #8961:
URL: https://github.com/apache/pinot/pull/8961#issuecomment-1180678011
@saurabhd336 - I can help reviewing this today. Can you take a look at the test failures ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on PR #8961:
URL: https://github.com/apache/pinot/pull/8961#issuecomment-1182554374
Suggest adding following tests -- Can be combined together where you generate some random data into a segment and then run queries on it or may be change any of the existing tests
- Segment generation with one or more raw SV columns with bloom filter
- Query execution tests on one or more raw SV columns with bloom filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8961:
URL: https://github.com/apache/pinot/pull/8961#discussion_r919464331
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -136,17 +138,132 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
bloomFilterConfig);
- try (BloomFilterCreator bloomFilterCreator = indexCreatorProvider.newBloomFilterCreator(
- IndexCreationContext.builder().withIndexDir(indexDir).withColumnMetadata(columnMetadata)
- .build().forBloomFilter(bloomFilterConfig));
- Dictionary dictionary = getDictionaryReader(columnMetadata, segmentWriter)) {
- int length = dictionary.length();
- for (int i = 0; i < length; i++) {
- bloomFilterCreator.add(dictionary.getStringValue(i));
+
+ if (columnMetadata.hasDictionary()) {
+ try (BloomFilterCreator bloomFilterCreator = indexCreatorProvider.newBloomFilterCreator(
+ IndexCreationContext.builder().withIndexDir(indexDir).withColumnMetadata(columnMetadata)
+ .build().forBloomFilter(bloomFilterConfig));
+ Dictionary dictionary = getDictionaryReader(columnMetadata, segmentWriter)) {
+ int length = dictionary.length();
+ for (int i = 0; i < length; i++) {
+ bloomFilterCreator.add(dictionary.getStringValue(i));
+ }
+ bloomFilterCreator.seal();
+ }
+ } else {
+ int numDocs = columnMetadata.getTotalDocs();
+ try (BloomFilterCreator bloomFilterCreator = indexCreatorProvider.newBloomFilterCreator(
+ IndexCreationContext.builder().withIndexDir(indexDir).withColumnMetadata(columnMetadata)
+ .build().forBloomFilter(bloomFilterConfig));
+ ForwardIndexReader forwardIndexReader = LoaderUtils.getForwardIndexReader(segmentWriter, columnMetadata);
+ ForwardIndexReaderContext readerContext = forwardIndexReader.createContext()) {
+ if (columnMetadata.isSingleValue()) {
+ // SV
+ switch (columnMetadata.getDataType()) {
+ case INT:
+ for (int i = 0; i < numDocs; i++) {
+ bloomFilterCreator.add(Integer.toString(forwardIndexReader.getInt(i, readerContext)));
+ }
+ break;
+ case LONG:
+ for (int i = 0; i < numDocs; i++) {
+ bloomFilterCreator.add(Long.toString(forwardIndexReader.getLong(i, readerContext)));
+ }
+ break;
+ case FLOAT:
+ for (int i = 0; i < numDocs; i++) {
+ bloomFilterCreator.add(Float.toString(forwardIndexReader.getFloat(i, readerContext)));
+ }
+ break;
+ case DOUBLE:
+ for (int i = 0; i < numDocs; i++) {
+ bloomFilterCreator.add(Double.toString(forwardIndexReader.getDouble(i, readerContext)));
+ }
+ break;
+ case STRING:
+ for (int i = 0; i < numDocs; i++) {
+ bloomFilterCreator.add(forwardIndexReader.getString(i, readerContext));
+ }
+ break;
+ case BYTES:
+ for (int i = 0; i < numDocs; i++) {
+ bloomFilterCreator.add(
+ BytesUtils.toHexString(forwardIndexReader.getBytes(i, readerContext))
+ );
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + columnMetadata.getDataType() + " for column: "
+ + columnMetadata.getColumnName());
+ }
+ bloomFilterCreator.seal();
Review Comment:
can be called exactly once after the if branch for dict or no-dict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
saurabhd336 commented on PR #8961:
URL: https://github.com/apache/pinot/pull/8961#issuecomment-1195533920
After further testing and discussions, came to realise that people generally move to no dictionary columns if they see high dictionary sizes. Introducing a hasSet to store unique values even for non dictionary columns, just to be able to support bloom filter (This PR was using the set just to provide the cardinality value when creating bloom filter) will be counterproductive. I'm closing this PR. Will rethink and raise a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 closed pull request #8961: Allow bloom filter for no dictionary SV columns
Posted by GitBox <gi...@apache.org>.
saurabhd336 closed pull request #8961: Allow bloom filter for no dictionary SV columns
URL: https://github.com/apache/pinot/pull/8961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org