You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/12/02 01:01:27 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #9896: In Kafka consumer, seek offset only when needed
Jackie-Jiang opened a new pull request, #9896:
URL: https://github.com/apache/pinot/pull/9896
Currently we seek offset for each fetch call, which adds unnecessary overhead, and also cause very noisy logs from `KafkaConsumer`.
Ideally we should only seek offset once when setting up the consumer, but to be more error prune, in this PR we track the last fetched offset, and seek the offset when the next fetch doesn't follow the offset from the previous fetch.
Also improve the filtering logic to only process the message when it should be kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9896: In Kafka consumer, seek offset only when needed
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9896:
URL: https://github.com/apache/pinot/pull/9896#issuecomment-1334659539
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9896?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9896](https://codecov.io/gh/apache/pinot/pull/9896?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (683495c) into [master](https://codecov.io/gh/apache/pinot/commit/716797fca1f4d140fb6244c7676b3d8a4414a57c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (716797f) will **increase** coverage by `0.10%`.
> The diff coverage is `42.85%`.
```diff
@@ Coverage Diff @@
## master #9896 +/- ##
============================================
+ Coverage 24.94% 25.05% +0.10%
Complexity 44 44
============================================
Files 1966 1966
Lines 105835 105841 +6
Branches 16056 16058 +2
============================================
+ Hits 26404 26519 +115
+ Misses 76743 76607 -136
- Partials 2688 2715 +27
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `25.05% <42.85%> (+0.10%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9896?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...in/stream/kafka20/KafkaPartitionLevelConsumer.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthUGFydGl0aW9uTGV2ZWxDb25zdW1lci5qYXZh) | `63.88% <42.85%> (-2.78%)` | :arrow_down: |
| [...ache/pinot/core/operator/docidsets/OrDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvT3JEb2NJZFNldC5qYXZh) | `56.81% <0.00%> (-43.19%)` | :arrow_down: |
| [...aggregation/groupby/ObjectGroupByResultHolder.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9ncm91cGJ5L09iamVjdEdyb3VwQnlSZXN1bHRIb2xkZXIuamF2YQ==) | `32.14% <0.00%> (-25.00%)` | :arrow_down: |
| [...nction/DistinctCountBitmapAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9EaXN0aW5jdENvdW50Qml0bWFwQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `39.89% <0.00%> (-9.85%)` | :arrow_down: |
| [.../pinot/common/function/scalar/ObjectFunctions.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL09iamVjdEZ1bmN0aW9ucy5qYXZh) | `7.69% <0.00%> (-7.70%)` | :arrow_down: |
| [...che/pinot/core/operator/docidsets/AndDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQW5kRG9jSWRTZXQuamF2YQ==) | `88.57% <0.00%> (-7.15%)` | :arrow_down: |
| [...core/startree/operator/StarTreeFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9vcGVyYXRvci9TdGFyVHJlZUZpbHRlck9wZXJhdG9yLmphdmE=) | `81.45% <0.00%> (-5.97%)` | :arrow_down: |
| [...core/operator/dociditerators/NotDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9Ob3REb2NJZEl0ZXJhdG9yLmphdmE=) | `95.00% <0.00%> (-5.00%)` | :arrow_down: |
| [...operator/query/LinearSelectionOrderByOperator.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9MaW5lYXJTZWxlY3Rpb25PcmRlckJ5T3BlcmF0b3IuamF2YQ==) | `65.38% <0.00%> (-4.62%)` | :arrow_down: |
| [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `52.70% <0.00%> (-2.71%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/apache/pinot/pull/9896/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #9896: In Kafka consumer, seek offset only when needed
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #9896:
URL: https://github.com/apache/pinot/pull/9896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org