You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/09/27 22:16:47 UTC
[GitHub] [pinot] cbalci opened a new pull request, #9475: [pinot-spark-connector] Bump spark connector max inbound message size
cbalci opened a new pull request, #9475:
URL: https://github.com/apache/pinot/pull/9475
Updating the GRPC client configuration used by the Pinot Spark Connector to use `Int.MaxValue` for message size which effectively bumps the max message size from 4MB to ~2GBs.
Spark-Connector is intended for potentially large reads, so 4MB can be a bottleneck for certain use cases (very wide tables).
Testing:
I've tested this locally with chunk sizes exceeding 4MB. I'm not adding an explicit test case not to unnecessarily burden the test runner.
`bugfix` `performance`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #9475: [pinot-spark-connector] Bump spark connector max inbound message size
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #9475:
URL: https://github.com/apache/pinot/pull/9475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9475: [pinot-spark-connector] Bump spark connector max inbound message size
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9475:
URL: https://github.com/apache/pinot/pull/9475#issuecomment-1260173345
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9475?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9475](https://codecov.io/gh/apache/pinot/pull/9475?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (89982f5) into [master](https://codecov.io/gh/apache/pinot/commit/d4392f9a60294f95b7001079322d96037defe799?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d4392f9) will **increase** coverage by `11.24%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #9475 +/- ##
=============================================
+ Coverage 25.90% 37.14% +11.24%
- Complexity 44 198 +154
=============================================
Files 1890 1911 +21
Lines 101219 101895 +676
Branches 15383 15456 +73
=============================================
+ Hits 26220 37849 +11629
+ Misses 72344 60906 -11438
- Partials 2655 3140 +485
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `25.97% <ø> (+0.07%)` | :arrow_up: |
| integration2 | `24.74% <ø> (?)` | |
| unittests2 | `15.50% <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9475?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inion/event/DefaultMinionEventObserverFactory.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXZlbnQvRGVmYXVsdE1pbmlvbkV2ZW50T2JzZXJ2ZXJGYWN0b3J5LmphdmE=) | `50.00% <0.00%> (-16.67%)` | :arrow_down: |
| [.../filter/predicate/InPredicateEvaluatorFactory.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL0luUHJlZGljYXRlRXZhbHVhdG9yRmFjdG9yeS5qYXZh) | `40.80% <0.00%> (-4.81%)` | :arrow_down: |
| [...ot/core/query/reduce/DistinctDataTableReducer.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvRGlzdGluY3REYXRhVGFibGVSZWR1Y2VyLmphdmE=) | `72.72% <0.00%> (-2.89%)` | :arrow_down: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `58.49% <0.00%> (-1.89%)` | :arrow_down: |
| [...ery/optimizer/filter/NumericalFilterOptimizer.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvZmlsdGVyL051bWVyaWNhbEZpbHRlck9wdGltaXplci5qYXZh) | `48.31% <0.00%> (-1.69%)` | :arrow_down: |
| [...t/processing/framework/SegmentProcessorConfig.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3Byb2Nlc3NpbmcvZnJhbWV3b3JrL1NlZ21lbnRQcm9jZXNzb3JDb25maWcuamF2YQ==) | `74.50% <0.00%> (-1.58%)` | :arrow_down: |
| [...core/operator/filter/predicate/PredicateUtils.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL1ByZWRpY2F0ZVV0aWxzLmphdmE=) | `51.51% <0.00%> (-1.52%)` | :arrow_down: |
| [...core/operator/docvalsets/TransformBlockValSet.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2N2YWxzZXRzL1RyYW5zZm9ybUJsb2NrVmFsU2V0LmphdmE=) | `35.84% <0.00%> (-1.05%)` | :arrow_down: |
| [...core/query/executor/ServerQueryExecutorV1Impl.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9leGVjdXRvci9TZXJ2ZXJRdWVyeUV4ZWN1dG9yVjFJbXBsLmphdmE=) | `76.53% <0.00%> (-0.85%)` | :arrow_down: |
| [...r/transform/function/LiteralTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vTGl0ZXJhbFRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `36.53% <0.00%> (-0.36%)` | :arrow_down: |
| ... and [529 more](https://codecov.io/gh/apache/pinot/pull/9475/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org