You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/02/04 19:16:04 UTC

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5051: Add forceDelete option for task queue deletion

Jackie-Jiang opened a new pull request #5051: Add forceDelete option for task queue deletion
URL: https://github.com/apache/incubator-pinot/pull/5051
 
 
   By default, force delete is set to false as it can corrupt the Helix
   Task Framework cache. We keep this as an expert only option to force
   deleting the task queue without going through the Helix transitions
   in case we need to perform some cleanups. In normal case, this flag
   should always be false.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io commented on issue #5051: Add forceDelete option for task queue deletion

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #5051: Add forceDelete option for task queue deletion
URL: https://github.com/apache/incubator-pinot/pull/5051#issuecomment-582085665
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=h1) Report
   > Merging [#5051](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/ec0da530f6c581c5cf56dd5ea566d2cd6390a815?src=pr&el=desc) will **decrease** coverage by `19.04%`.
   > The diff coverage is `42.85%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/5051/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=tree)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #5051       +/-   ##
   =============================================
   - Coverage     58.54%   39.49%   -19.05%     
   =============================================
     Files          1182     1182               
     Lines         63135    63135               
     Branches       9287     9288        +1     
   =============================================
   - Hits          36963    24937    -12026     
   - Misses        23455    36053    +12598     
   + Partials       2717     2145      -572
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...roller/api/resources/PinotTaskRestletResource.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFza1Jlc3RsZXRSZXNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...lix/core/minion/PinotHelixTaskResourceManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdEhlbGl4VGFza1Jlc291cmNlTWFuYWdlci5qYXZh) | `82.95% <60%> (+9.69%)` | `0 <0> (ø)` | :arrow_down: |
   | [.../org/apache/pinot/common/http/MultiGetRequest.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vaHR0cC9NdWx0aUdldFJlcXVlc3QuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | [...che/pinot/common/restlet/resources/TablesList.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzdGxldC9yZXNvdXJjZXMvVGFibGVzTGlzdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | [...che/pinot/pql/parsers/pql2/ast/OptionsAstNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9wcWwvcGFyc2Vycy9wcWwyL2FzdC9PcHRpb25zQXN0Tm9kZS5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | [...ot/core/query/scheduler/TableBasedGroupMapper.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvVGFibGVCYXNlZEdyb3VwTWFwcGVyLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | [...t/core/segment/index/readers/OnHeapDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L3JlYWRlcnMvT25IZWFwRGljdGlvbmFyeS5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | [...t/core/query/scheduler/OutOfCapacityException.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvT3V0T2ZDYXBhY2l0eUV4Y2VwdGlvbi5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | [...ot/core/query/scheduler/SchedulerQueryContext.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvU2NoZWR1bGVyUXVlcnlDb250ZXh0LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | [...pinot/common/exception/TableNotFoundException.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1RhYmxlTm90Rm91bmRFeGNlcHRpb24uamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (ø)` | |
   | ... and [562 more](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=footer). Last update [ec0da53...b1cae6f](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5051: Add forceDelete option for task queue deletion

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #5051: Add forceDelete option for task queue deletion
URL: https://github.com/apache/incubator-pinot/pull/5051
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5051: Add forceDelete option for task queue deletion

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #5051: Add forceDelete option for task queue deletion
URL: https://github.com/apache/incubator-pinot/pull/5051#discussion_r374873925
 
 

 ##########
 File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SimpleMinionClusterIntegrationTest.java
 ##########
 @@ -174,10 +173,12 @@ public void testStopAndResumeTaskQueue()
     }, 60_000L, "Failed to get all tasks COMPLETED");
 
     // Delete the task queue
-    // Note: Comment out the api for now since there is a known race condition
-    // where helix controller might write a deleted workflow back to ZK because it's still caching it.
-    // TODO: revert this after merging the fix
-//    _helixTaskResourceManager.deleteTaskQueue(TestTaskGenerator.TASK_TYPE);
+    _helixTaskResourceManager.deleteTaskQueue(TestTaskGenerator.TASK_TYPE, false);
+
+    // Wait at most 60 seconds for task queue to be deleted
+    TestUtils.waitForCondition(input -> {
+      return !_helixTaskResourceManager.getTaskTypes().contains(TestTaskGenerator.TASK_TYPE);
+    }, 60_000L, "Failed to delete the task queue");
 
 Review comment:
   Extract the timeout and make it as a constant.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io edited a comment on issue #5051: Add forceDelete option for task queue deletion

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #5051: Add forceDelete option for task queue deletion
URL: https://github.com/apache/incubator-pinot/pull/5051#issuecomment-582085665
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=h1) Report
   > Merging [#5051](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/ec0da530f6c581c5cf56dd5ea566d2cd6390a815?src=pr&el=desc) will **increase** coverage by `0.02%`.
   > The diff coverage is `42.85%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/5051/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #5051      +/-   ##
   ============================================
   + Coverage     58.54%   58.57%   +0.02%     
     Complexity       12       12              
   ============================================
     Files          1182     1182              
     Lines         63135    63135              
     Branches       9287     9288       +1     
   ============================================
   + Hits          36963    36981      +18     
   + Misses        23455    23434      -21     
   - Partials       2717     2720       +3
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...roller/api/resources/PinotTaskRestletResource.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFza1Jlc3RsZXRSZXNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...lix/core/minion/PinotHelixTaskResourceManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdEhlbGl4VGFza1Jlc291cmNlTWFuYWdlci5qYXZh) | `82.95% <60%> (+9.69%)` | `0 <0> (ø)` | :arrow_down: |
   | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `50% <0%> (-25%)` | `0% <0%> (ø)` | |
   | [...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `53.76% <0%> (-11.83%)` | `0% <0%> (ø)` | |
   | [...r/dociditerators/RangelessBitmapDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9SYW5nZWxlc3NCaXRtYXBEb2NJZEl0ZXJhdG9yLmphdmE=) | `72.22% <0%> (-11.12%)` | `0% <0%> (ø)` | |
   | [...core/operator/dociditerators/AndDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9BbmREb2NJZEl0ZXJhdG9yLmphdmE=) | `81.03% <0%> (-6.9%)` | `0% <0%> (ø)` | |
   | [...e/operator/dociditerators/SortedDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9Tb3J0ZWREb2NJZEl0ZXJhdG9yLmphdmE=) | `55.55% <0%> (-5.56%)` | `0% <0%> (ø)` | |
   | [...e/operator/dociditerators/SVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9TVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `65.51% <0%> (-4.6%)` | `0% <0%> (ø)` | |
   | [...troller/helix/core/retention/RetentionManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JldGVudGlvbi9SZXRlbnRpb25NYW5hZ2VyLmphdmE=) | `75% <0%> (-4.17%)` | `0% <0%> (ø)` | |
   | [...rg/apache/pinot/core/operator/CombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9Db21iaW5lT3BlcmF0b3IuamF2YQ==) | `66.29% <0%> (-2.25%)` | `0% <0%> (ø)` | |
   | ... and [18 more](https://codecov.io/gh/apache/incubator-pinot/pull/5051/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=footer). Last update [ec0da53...b1cae6f](https://codecov.io/gh/apache/incubator-pinot/pull/5051?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org