You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/03/11 21:42:10 UTC
[GitHub] [incubator-pinot] npawar opened a new pull request #5143: Fix
hybrid quickstart
npawar opened a new pull request #5143: Fix hybrid quickstart
URL: https://github.com/apache/incubator-pinot/pull/5143
Hybrid quickstart consumes a lot of cpu and memory as soon it starts up and it keeps on increasing. If another hybrid table is created on it, it eventually runs out of memory and crashes. Possible reasons:
1. we're creating 2 servers, 2 brokers, which each take up a lot of resources by default
2. the airlineStats stream produces events continuously in a loop, with no rate limiting.
Fixes:
1. Creating just 1 server and 1 broker. This was initially designed with 2 because we didn't have support for the same server to be tagged with _OFFLINE and _REALTIME, which no longer holds true
2. Sleeping for 1 second after each event published to the stream, creating a steady ingestion rate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar merged pull request #5143: Fix hybrid
quickstart
Posted by GitBox <gi...@apache.org>.
npawar merged pull request #5143: Fix hybrid quickstart
URL: https://github.com/apache/incubator-pinot/pull/5143
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on issue #5143: Fix hybrid
quickstart
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #5143: Fix hybrid quickstart
URL: https://github.com/apache/incubator-pinot/pull/5143#issuecomment-597924254
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=h1) Report
> Merging [#5143](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/46698d8c78e93afa579ff0b968a7eb291090503d&el=desc) will **decrease** coverage by `21.37%`.
> The diff coverage is `53.33%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #5143 +/- ##
=============================================
- Coverage 65.67% 44.29% -21.38%
=============================================
Files 1051 1052 +1
Lines 54125 54136 +11
Branches 8070 8080 +10
=============================================
- Hits 35546 23979 -11567
- Misses 15965 28091 +12126
+ Partials 2614 2066 -548
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...t/index/readers/ConstantValueStringDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZVN0cmluZ0RpY3Rpb25hcnkuamF2YQ==) | `37.50% <16.66%> (-12.50%)` | `0.00 <0.00> (ø)` | |
| [...nt/index/readers/ConstantValueBytesDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZUJ5dGVzRGljdGlvbmFyeS5qYXZh) | `37.50% <37.50%> (ø)` | `0.00 <0.00> (?)` | |
| [...t/index/readers/ConstantValueDoubleDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZURvdWJsZURpY3Rpb25hcnkuamF2YQ==) | `53.33% <37.50%> (+13.33%)` | `0.00 <0.00> (ø)` | |
| [...nt/index/readers/ConstantValueFloatDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZUZsb2F0RGljdGlvbmFyeS5qYXZh) | `53.33% <37.50%> (+3.33%)` | `0.00 <0.00> (ø)` | |
| [...ment/index/readers/ConstantValueIntDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZUludERpY3Rpb25hcnkuamF2YQ==) | `53.33% <44.44%> (+3.33%)` | `0.00 <0.00> (ø)` | |
| [...ent/index/readers/ConstantValueLongDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZUxvbmdEaWN0aW9uYXJ5LmphdmE=) | `53.33% <44.44%> (+3.33%)` | `0.00 <0.00> (ø)` | |
| [...t/core/io/reader/impl/ConstantMVInvertedIndex.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pby9yZWFkZXIvaW1wbC9Db25zdGFudE1WSW52ZXJ0ZWRJbmRleC5qYXZh) | `57.14% <57.14%> (ø)` | `0.00 <0.00> (?)` | |
| [...not/core/io/reader/impl/ConstantSVSortedIndex.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pby9yZWFkZXIvaW1wbC9Db25zdGFudFNWU29ydGVkSW5kZXguamF2YQ==) | `54.54% <60.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `79.31% <61.53%> (-2.96%)` | `0.00 <0.00> (ø)` | |
| [...ot/core/io/reader/impl/ConstantMVForwardIndex.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pby9yZWFkZXIvaW1wbC9Db25zdGFudE1WRm9yd2FyZEluZGV4LmphdmE=) | `57.14% <66.66%> (ø)` | `0.00 <0.00> (?)` | |
| ... and [567 more](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=footer). Last update [46698d8...e41e5a0](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on issue #5143: Fix
hybrid quickstart
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #5143: Fix hybrid quickstart
URL: https://github.com/apache/incubator-pinot/pull/5143#issuecomment-597924254
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=h1) Report
> Merging [#5143](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/46698d8c78e93afa579ff0b968a7eb291090503d&el=desc) will **decrease** coverage by `9.18%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #5143 +/- ##
============================================
- Coverage 65.67% 56.48% -9.19%
Complexity 12 12
============================================
Files 1051 1052 +1
Lines 54125 54136 +11
Branches 8070 8080 +10
============================================
- Hits 35546 30580 -4966
- Misses 15965 21114 +5149
+ Partials 2614 2442 -172
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...ot/minion/events/EventObserverFactoryRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXZlbnRzL0V2ZW50T2JzZXJ2ZXJGYWN0b3J5UmVnaXN0cnkuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...t/minion/executor/TaskExecutorFactoryRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhlY3V0b3IvVGFza0V4ZWN1dG9yRmFjdG9yeVJlZ2lzdHJ5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...minion/executor/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhlY3V0b3IvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...nion/events/DefaultMinionEventObserverFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXZlbnRzL0RlZmF1bHRNaW5pb25FdmVudE9ic2VydmVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...startree/executor/StarTreeAggregationExecutor.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9leGVjdXRvci9TdGFyVHJlZUFnZ3JlZ2F0aW9uRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...executor/ConvertToRawIndexTaskExecutorFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhlY3V0b3IvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3JGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| ... and [310 more](https://codecov.io/gh/apache/incubator-pinot/pull/5143/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=footer). Last update [46698d8...e41e5a0](https://codecov.io/gh/apache/incubator-pinot/pull/5143?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org