You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/30 22:41:51 UTC
[GitHub] [pinot] priyen opened a new pull request, #9883: Distinct Count HLL pre-aggregation in realtime segments
priyen opened a new pull request, #9883:
URL: https://github.com/apache/pinot/pull/9883
This adds support for distinct count hll pre-aggregation. It introduces a new property on the fieldSpec, fixedLength in bytes so that BYTES data type can be treated as fixed length and we can utilize the FixedByteSVMutableForwardIndex.
When used for Hyperloglog data values, the fixedLength should represent in bytes the size of the Hyperloglog object when serialized.
Hyperloglog w/ log2m of 8 has a size of 180 bytes, with a log2m of 12 has a size of 2740 bytes. I unit tested using log2m of 12 because that's the size one of our use cases require
unit tests for the fixedByte mutable forward indexes' getBytes() and setBytes() new implementation
unit tests for aggregating rows and asserting on their Hyperloglog objects
Instructions:
The PR has to be tagged with at least one of the following labels (*):
feature
bugfix
performance
ui
backward-incompat
release-notes (**)
Remove these instructions before publishing the PR.
(*) Other labels to consider:
testing
dependencies
docker
kubernetes
observability
security
code-style
extension-point
refactor
cleanup
(**) Use release-notes label for scenarios like:
New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9883: Distinct Count HLL pre-aggregation in realtime segments
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9883:
URL: https://github.com/apache/pinot/pull/9883#issuecomment-1334065605
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9883?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9883](https://codecov.io/gh/apache/pinot/pull/9883?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4ea68e9) into [master](https://codecov.io/gh/apache/pinot/commit/c8c6b253ae8f0a8e9e3293b99bd37d946f94a19d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c8c6b25) will **decrease** coverage by `43.47%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #9883 +/- ##
=============================================
- Coverage 68.57% 25.10% -43.48%
+ Complexity 4979 44 -4935
=============================================
Files 1952 1966 +14
Lines 104809 105903 +1094
Branches 15866 16061 +195
=============================================
- Hits 71876 26587 -45289
- Misses 27856 76609 +48753
+ Partials 5077 2707 -2370
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `25.10% <0.00%> (?)` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9883?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...al/aggregator/DistinctCountHLLValueAggregator.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9hZ2dyZWdhdG9yL0Rpc3RpbmN0Q291bnRITExWYWx1ZUFnZ3JlZ2F0b3IuamF2YQ==) | `0.00% <0.00%> (-85.72%)` | :arrow_down: |
| [...t/segment/local/aggregator/SumValueAggregator.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9hZ2dyZWdhdG9yL1N1bVZhbHVlQWdncmVnYXRvci5qYXZh) | `0.00% <0.00%> (-72.73%)` | :arrow_down: |
| [...gment/local/aggregator/ValueAggregatorFactory.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9hZ2dyZWdhdG9yL1ZhbHVlQWdncmVnYXRvckZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (-84.62%)` | :arrow_down: |
| [...exsegment/mutable/DefaultMutableIndexProvider.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9EZWZhdWx0TXV0YWJsZUluZGV4UHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (-85.37%)` | :arrow_down: |
| [...local/indexsegment/mutable/MutableSegmentImpl.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9NdXRhYmxlU2VnbWVudEltcGwuamF2YQ==) | `0.00% <0.00%> (-57.76%)` | :arrow_down: |
| [...e/impl/forward/FixedByteSVMutableForwardIndex.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ZvcndhcmQvRml4ZWRCeXRlU1ZNdXRhYmxlRm9yd2FyZEluZGV4LmphdmE=) | `0.00% <0.00%> (-93.88%)` | :arrow_down: |
| [...cal/startree/v2/builder/BaseSingleTreeBuilder.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9idWlsZGVyL0Jhc2VTaW5nbGVUcmVlQnVpbGRlci5qYXZh) | `0.00% <0.00%> (-90.00%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/TableConfigUtils.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9UYWJsZUNvbmZpZ1V0aWxzLmphdmE=) | `0.00% <0.00%> (-68.16%)` | :arrow_down: |
| [...main/java/org/apache/pinot/spi/data/FieldSpec.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9GaWVsZFNwZWMuamF2YQ==) | `0.00% <ø> (-81.34%)` | :arrow_down: |
| [...in/java/org/apache/pinot/spi/utils/BytesUtils.java](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQnl0ZXNVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1515 more](https://codecov.io/gh/apache/pinot/pull/9883/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] priyen closed pull request #9883: Distinct Count HLL pre-aggregation in realtime segments
Posted by "priyen (via GitHub)" <gi...@apache.org>.
priyen closed pull request #9883: Distinct Count HLL pre-aggregation in realtime segments
URL: https://github.com/apache/pinot/pull/9883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org