You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/04/15 18:16:50 UTC
[GitHub] [pinot] klsince opened a new pull request, #8551: endpoints to get and delete minion task metadata
klsince opened a new pull request, #8551:
URL: https://github.com/apache/pinot/pull/8551
add two minion tasks related endpoints to allow GET and DELETE /tasks/{taskType}/{tableNameWithType}/metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #8551: endpoints to get and delete minion task metadata
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #8551:
URL: https://github.com/apache/pinot/pull/8551#discussion_r851568448
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java:
##########
@@ -149,6 +149,28 @@ public Map<String, TaskState> getTaskStatesByTable(
return _pinotHelixTaskResourceManager.getTaskStatesByTable(taskType, tableNameWithType);
}
+ @GET
+ @Path("/tasks/{taskType}/{tableNameWithType}/metadata")
+ @ApiOperation("Get task metadata for the given task type and table")
+ public String getTaskMetadataByTable(
+ @ApiParam(value = "Task type", required = true) @PathParam("taskType") String taskType,
+ @ApiParam(value = "Table name with type", required = true) @PathParam("tableNameWithType")
+ String tableNameWithType) {
+ return _pinotHelixTaskResourceManager.getTaskMetadataByTable(taskType, tableNameWithType);
+ }
+
+ @DELETE
Review Comment:
Helix task framework is not aware of the task metadata here, e.g. RealtimToOfflineTask keeps a timestamp as watermark in a ZNode under `/Propertystore/MINION_TASK_METADATA/` to manage its progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8551: endpoints to get and delete minion task metadata
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8551:
URL: https://github.com/apache/pinot/pull/8551#issuecomment-1100425267
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8551?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8551](https://codecov.io/gh/apache/pinot/pull/8551?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (89bf429) into [master](https://codecov.io/gh/apache/pinot/commit/05a7f28b7316a35b171637c3be836345e4ebbe74?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (05a7f28) will **decrease** coverage by `0.09%`.
> The diff coverage is `22.22%`.
```diff
@@ Coverage Diff @@
## master #8551 +/- ##
============================================
- Coverage 70.77% 70.68% -0.10%
Complexity 4290 4290
============================================
Files 1681 1681
Lines 87957 87971 +14
Branches 13320 13321 +1
============================================
- Hits 62252 62181 -71
- Misses 21354 21440 +86
+ Partials 4351 4350 -1
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `27.04% <22.22%> (-0.17%)` | :arrow_down: |
| integration2 | `25.93% <22.22%> (+0.03%)` | :arrow_up: |
| unittests1 | `67.02% <ø> (-0.01%)` | :arrow_down: |
| unittests2 | `14.05% <16.66%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...roller/api/resources/PinotTaskRestletResource.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFza1Jlc3RsZXRSZXNvdXJjZS5qYXZh) | `2.83% <0.00%> (-0.09%)` | :arrow_down: |
| [...lix/core/minion/PinotHelixTaskResourceManager.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdEhlbGl4VGFza1Jlc291cmNlTWFuYWdlci5qYXZh) | `39.80% <23.07%> (-1.97%)` | :arrow_down: |
| [...apache/pinot/controller/BaseControllerStarter.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9CYXNlQ29udHJvbGxlclN0YXJ0ZXIuamF2YQ==) | `82.33% <100.00%> (ø)` | |
| [...data/manager/realtime/DefaultSegmentCommitter.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvRGVmYXVsdFNlZ21lbnRDb21taXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-80.00%)` | :arrow_down: |
| [...ller/helix/core/minion/TaskTypeMetricsUpdater.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9UYXNrVHlwZU1ldHJpY3NVcGRhdGVyLmphdmE=) | `80.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...er/api/resources/LLCSegmentCompletionHandlers.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL0xMQ1NlZ21lbnRDb21wbGV0aW9uSGFuZGxlcnMuamF2YQ==) | `43.56% <0.00%> (-18.82%)` | :arrow_down: |
| [...data/manager/realtime/SegmentCommitterFactory.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VnbWVudENvbW1pdHRlckZhY3RvcnkuamF2YQ==) | `88.23% <0.00%> (-11.77%)` | :arrow_down: |
| [...ache/pinot/core/operator/docidsets/OrDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvT3JEb2NJZFNldC5qYXZh) | `86.36% <0.00%> (-11.37%)` | :arrow_down: |
| [...or/transform/function/IsNullTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSXNOdWxsVHJhbnNmb3JtRnVuY3Rpb24uamF2YQ==) | `67.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `51.42% <0.00%> (-6.67%)` | :arrow_down: |
| ... and [13 more](https://codecov.io/gh/apache/pinot/pull/8551/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8551?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8551?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [05a7f28...89bf429](https://codecov.io/gh/apache/pinot/pull/8551?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #8551: endpoints to get and delete minion task metadata
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #8551:
URL: https://github.com/apache/pinot/pull/8551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on a diff in pull request #8551: endpoints to get and delete minion task metadata
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on code in PR #8551:
URL: https://github.com/apache/pinot/pull/8551#discussion_r851555399
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotHelixTaskResourceManager.java:
##########
@@ -643,6 +650,23 @@ public String getParentTaskName(String taskType, String taskName) {
return TASK_PREFIX + taskType + TASK_NAME_SEPARATOR + taskName;
}
+ public String getTaskMetadataByTable(String taskType, String tableNameWithType) {
+ ZkHelixPropertyStore<ZNRecord> propertyStore = _helixResourceManager.getPropertyStore();
+ ZNRecord raw = MinionTaskMetadataUtils.fetchTaskMetadata(propertyStore, taskType, tableNameWithType);
+ Preconditions
+ .checkState(raw != null, "Found task metadata for task type: %s for table: %s", taskType, tableNameWithType);
+ try {
+ return JsonUtils.objectToString(raw);
+ } catch (JsonProcessingException e) {
+ throw new RuntimeException("Failed to convert task metadata to Json format", e);
Review Comment:
should this be a 5xx return?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotHelixTaskResourceManager.java:
##########
@@ -643,6 +650,23 @@ public String getParentTaskName(String taskType, String taskName) {
return TASK_PREFIX + taskType + TASK_NAME_SEPARATOR + taskName;
}
+ public String getTaskMetadataByTable(String taskType, String tableNameWithType) {
+ ZkHelixPropertyStore<ZNRecord> propertyStore = _helixResourceManager.getPropertyStore();
+ ZNRecord raw = MinionTaskMetadataUtils.fetchTaskMetadata(propertyStore, taskType, tableNameWithType);
+ Preconditions
Review Comment:
so, if the metadata is not there, we return 5xx? Please change it to return a 4xx.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java:
##########
@@ -149,6 +149,28 @@ public Map<String, TaskState> getTaskStatesByTable(
return _pinotHelixTaskResourceManager.getTaskStatesByTable(taskType, tableNameWithType);
}
+ @GET
+ @Path("/tasks/{taskType}/{tableNameWithType}/metadata")
+ @ApiOperation("Get task metadata for the given task type and table")
+ public String getTaskMetadataByTable(
+ @ApiParam(value = "Task type", required = true) @PathParam("taskType") String taskType,
+ @ApiParam(value = "Table name with type", required = true) @PathParam("tableNameWithType")
+ String tableNameWithType) {
+ return _pinotHelixTaskResourceManager.getTaskMetadataByTable(taskType, tableNameWithType);
+ }
+
+ @DELETE
Review Comment:
helix deletes task metadata after sometime, right ? Why provide an endpoint to delete it? Have you verified with helix that this ok? and there is no race condition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org