You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/07/12 05:41:32 UTC
[GitHub] [pinot] saurabhd336 opened a new pull request, #9043: Task generator meta api
saurabhd336 opened a new pull request, #9043:
URL: https://github.com/apache/pinot/pull/9043
Controller API to return task generator metadata (followup PR for https://github.com/apache/pinot/pull/9011)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #9043: Task generator meta api + save task generator info to ZK
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1182656351
I am not for using ZK as a log sink. There are metrics, there are logs for each individual service, there are also ways of getting exceptions like we do on the server (save the last N exceptions and return with the status).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #9043: Task generator debug api
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1184649708
@saurabhd336 I cannot add comments to your design doc. Also, please close this PR and add your design doc to the issue (create one if there is none).
Thanks
If you are not modifying zk this may be simple enough that you don't need a design doc, but since you already have one, that is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on pull request #9043: Task generator meta api + save task generator info to ZK
Posted by GitBox <gi...@apache.org>.
npawar commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1182696307
Closing this, as the work is being picked by @saurabhd336 in https://github.com/apache/pinot/pull/9043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on pull request #9043: Task generator meta api + save task generator info to ZK
Posted by GitBox <gi...@apache.org>.
npawar commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1182584155
> Let us clear PR 9011 first
we might have to continue the discussion here instead, because the dev doing 9011 is on a long leave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on pull request #9043: Task generator debug api
Posted by GitBox <gi...@apache.org>.
saurabhd336 commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1184326651
Updated PR description with design doc detailing the goals and design for this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #9043: Task generator meta api + save task generator info to ZK
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1182655776
> > Let us clear PR 9011 first
>
> we might have to continue the discussion here instead, because the dev doing 9011 is on a long leave.
OK, then let us close the other PR, and continue the discussion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on pull request #9043: Task generator debug api
Posted by GitBox <gi...@apache.org>.
saurabhd336 commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1183388829
I've updated the PR to reflect the discussion. Mainly moved to an inmemory map for storing the debug data. If need be, we can easily move to ZK based storage by writing another implementation of the `TaskManagerStatusCache` interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on pull request #9043: Task generator meta api + save task generator info to ZK
Posted by GitBox <gi...@apache.org>.
npawar commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1182695192
> I am not for using ZK as a log sink. There are metrics, there are logs for each individual service, there are also ways of getting exceptions like we do on the server (save the last N exceptions and return with the status).
Yea keeping in memory (like consumer errors) sounds like a fine first approach. I'm also in favor of starting off with that, and moving to zk if need for persistence arises based on user feedback (or exploring the more elaborate solutions at that time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9043: Task generator meta api + save task generator info to ZK
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1181358067
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9043?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9043](https://codecov.io/gh/apache/pinot/pull/9043?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (abbc540) into [master](https://codecov.io/gh/apache/pinot/commit/0c55422d8bb5ff0df3e7871c9b1a1e52be38aecc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0c55422) will **decrease** coverage by `48.13%`.
> The diff coverage is `2.32%`.
```diff
@@ Coverage Diff @@
## master #9043 +/- ##
=============================================
- Coverage 63.49% 15.36% -48.14%
+ Complexity 4714 170 -4544
=============================================
Files 1783 1787 +4
Lines 94134 94258 +124
Branches 14140 14154 +14
=============================================
- Hits 59774 14486 -45288
- Misses 30102 78733 +48631
+ Partials 4258 1039 -3219
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `?` | |
| unittests2 | `15.36% <2.32%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9043?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ache/pinot/common/metadata/ZKMetadataProvider.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0YWRhdGEvWktNZXRhZGF0YVByb3ZpZGVyLmphdmE=) | `0.00% <0.00%> (-15.26%)` | :arrow_down: |
| [...che/pinot/common/minion/BaseTaskGeneratorInfo.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWluaW9uL0Jhc2VUYXNrR2VuZXJhdG9ySW5mby5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...he/pinot/common/minion/TaskGeneratorInfoUtils.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWluaW9uL1Rhc2tHZW5lcmF0b3JJbmZvVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [.../common/minion/TaskGeneratorMostRecentRunInfo.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWluaW9uL1Rhc2tHZW5lcmF0b3JNb3N0UmVjZW50UnVuSW5mby5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...on/minion/TaskGeneratorMostRecentRunInfoUtils.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWluaW9uL1Rhc2tHZW5lcmF0b3JNb3N0UmVjZW50UnVuSW5mb1V0aWxzLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...he/pinot/common/utils/helix/FakePropertyStore.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaGVsaXgvRmFrZVByb3BlcnR5U3RvcmUuamF2YQ==) | `0.00% <0.00%> (-56.53%)` | :arrow_down: |
| [...roller/api/resources/PinotTaskRestletResource.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFza1Jlc3RsZXRSZXNvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...lix/core/minion/PinotHelixTaskResourceManager.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdEhlbGl4VGFza1Jlc291cmNlTWFuYWdlci5qYXZh) | `2.40% <0.00%> (-0.02%)` | :arrow_down: |
| [...controller/helix/core/minion/PinotTaskManager.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdFRhc2tNYW5hZ2VyLmphdmE=) | `36.47% <11.53%> (-2.21%)` | :arrow_down: |
| [...src/main/java/org/apache/pinot/sql/FilterKind.java](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvRmlsdGVyS2luZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1160 more](https://codecov.io/gh/apache/pinot/pull/9043/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/9043?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/9043?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0c55422...abbc540](https://codecov.io/gh/apache/pinot/pull/9043?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on pull request #9043: Task generator debug api
Posted by GitBox <gi...@apache.org>.
saurabhd336 commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1184729359
I've raised a new PR https://github.com/apache/pinot/pull/9058.
Doc link has been fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 closed pull request #9043: Task generator debug api
Posted by GitBox <gi...@apache.org>.
saurabhd336 closed pull request #9043: Task generator debug api
URL: https://github.com/apache/pinot/pull/9043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #9043: Task generator debug api
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #9043:
URL: https://github.com/apache/pinot/pull/9043#issuecomment-1183402833
In general, use alerts and logs to debug problems. If you really want to persist problem state over time, the approach I prefer is to sprinkle code all over pinot (similar to metrics) that emit some events to a stream underneath. This stream can be fielded by Pinot into a pinot table, and used to debug the pinot cluster via queries: "How many minion jobs failed last week vs previous week?", "Which table has the most errors loading a segment?" etc.
@npawar, @kishoreg and I have discussed before about introducing new znodes for new features (let alone debugging), and came to the conclusion that it is hard to clean up things afterwards.
Anyways, since we are concluding to go with keeping stuff in memory and using APIs to debug, I suggest you put out an issue and add some interface definitions there (or a doc if you choose).
Please close thsis PR and begin a new one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org