You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/08/10 23:41:09 UTC

[GitHub] [pinot] sajjad-moradi opened a new pull request, #9197: Reset realtime consumption

sajjad-moradi opened a new pull request, #9197:
URL: https://github.com/apache/pinot/pull/9197

   ## Description
   For realtime segments, if there's any changes to stream configs in the table config, the changes don't get picked up immediately in the existing consuming segments. After the existing consuming segments complete - which may take hours depending on flush thresholds defined in table config - the new consuming segments will pick up the stream config changes.
   In this PR, the force commit functionality that was introduced in pause/resume feature (#8986) is used in a controller endpoint to reset the consumption of a realtime table. After "consumption reset" is issued, the current consuming segments will be forced to commit immediately and then the new consuming segments will pick up any changes in the table config.
   
   ## Testing Done
   Verified the expected behavior using `LLCRealtimeClusterIntegrationTest`. During ingestion, I added a consumption rate limit parameter to the table config and then used `resetConsumption` endpoint and verified that the existing consuming segments completed and then the new consuming segments consumed according to the specified consumption rate limit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #9197:
URL: https://github.com/apache/pinot/pull/9197#discussion_r956405466


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")
+  @ApiOperation(value = "Reset consumption of a realtime table",
+      notes = "Reset the consumption for a realtime table using force commit")

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9197:
URL: https://github.com/apache/pinot/pull/9197#discussion_r946206230


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")

Review Comment:
   Suggest making the API more explicit that it is just force committing the consuming segment. Something like `POST /tables/{tableName}/forceCommit`



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")
+  @ApiOperation(value = "Reset consumption of a realtime table",
+      notes = "Reset the consumption for a realtime table using force commit")
+  public Response resetConsumption(
+      @ApiParam(value = "Name of the table", required = true) @PathParam("tableName") String tableName) {

Review Comment:
   Suggest adding an optional segment so that we only force commit one segment?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi merged pull request #9197: Force commit consuming segments

Posted by GitBox <gi...@apache.org>.
sajjad-moradi merged PR #9197:
URL: https://github.com/apache/pinot/pull/9197


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9197:
URL: https://github.com/apache/pinot/pull/9197#issuecomment-1211430729

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/9197?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#9197](https://codecov.io/gh/apache/pinot/pull/9197?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e8de3a8) into [master](https://codecov.io/gh/apache/pinot/commit/56927a193a31b42e542aec75390a4a0bdfac774a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (56927a1) will **decrease** coverage by `13.16%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #9197       +/-   ##
   =============================================
   - Coverage     28.44%   15.27%   -13.17%     
   - Complexity       53      170      +117     
   =============================================
     Files          1840     1802       -38     
     Lines         98407    96594     -1813     
     Branches      14983    14771      -212     
   =============================================
   - Hits          27991    14756    -13235     
   - Misses        67731    80779    +13048     
   + Partials       2685     1059     -1626     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests2 | `15.27% <0.00%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9197?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ller/api/resources/PinotRealtimeTableResource.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90UmVhbHRpbWVUYWJsZVJlc291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [.../core/realtime/PinotLLCRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90TExDUmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `60.68% <0.00%> (+2.13%)` | :arrow_up: |
   | [...src/main/java/org/apache/pinot/sql/FilterKind.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvRmlsdGVyS2luZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/util/GroupByUtils.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL0dyb3VwQnlVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/common/auth/AuthConfig.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXV0aC9BdXRoQ29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/sql/parsers/PinotSqlType.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9QaW5vdFNxbFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/operator/BaseOperator.java](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9CYXNlT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [929 more](https://codecov.io/gh/apache/pinot/pull/9197/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] KKcorps commented on a diff in pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
KKcorps commented on code in PR #9197:
URL: https://github.com/apache/pinot/pull/9197#discussion_r944229173


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")
+  @ApiOperation(value = "Reset consumption of a realtime table",
+      notes = "Reset the consumption for a realtime table using force commit")

Review Comment:
   ```suggestion
         notes = "Force commit the current segments in consuming state and restart consumption. This should be used after schema/table config changes")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #9197:
URL: https://github.com/apache/pinot/pull/9197#discussion_r956439015


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")
+  @ApiOperation(value = "Reset consumption of a realtime table",
+      notes = "Reset the consumption for a realtime table using force commit")

Review Comment:
   For the status of forceCommit, I can't think of an easy way to figure out if a segment is committed after force commit!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a diff in pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #9197:
URL: https://github.com/apache/pinot/pull/9197#discussion_r947039545


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")

Review Comment:
   +1 . we already have reset API which does helix disable/enable. this is different in logic, so calling it forceCommit would avoid confusion



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")
+  @ApiOperation(value = "Reset consumption of a realtime table",
+      notes = "Reset the consumption for a realtime table using force commit")

Review Comment:
   Maybe add some note in the description about it being async, and 200 response may not mean it has actually been done already? Will there be a followup to this to know status of resetConsumption (like the recent reload status API)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #9197:
URL: https://github.com/apache/pinot/pull/9197#discussion_r956407470


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")
+  @ApiOperation(value = "Reset consumption of a realtime table",
+      notes = "Reset the consumption for a realtime table using force commit")
+  public Response resetConsumption(
+      @ApiParam(value = "Name of the table", required = true) @PathParam("tableName") String tableName) {

Review Comment:
   We don't have partition level pausing. I know this endpoint is a bit different than pause/resume, but let me think about it a bit and maybe we can come up with a way for partition leve operation on all endpoints in this class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9197: Reset realtime consumption

Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #9197:
URL: https://github.com/apache/pinot/pull/9197#discussion_r956403213


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java:
##########
@@ -90,6 +90,23 @@ public Response resumeConsumption(
     }
   }
 
+  @POST
+  @Path("/tables/{tableName}/resetConsumption")

Review Comment:
   Valid point. Refactored.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org