[GitHub] [samza] lakshmi-manasa-g opened a new pull request, #1607: SAMZA-2734: [Elasticity] Update last processed offset after an envelopepe is finished processing when elasticity is enabled

[GitBox <gi...@apache.org>]

lakshmi-manasa-g opened a new pull request, #1607:
URL: https://github.com/apache/samza/pull/1607

   **Feature:** Elasticity (SAMZA-2687) for a Samza job allows job to have more tasks than the number of input SystemStreamPartition(SSP). Thus, a job can scale up beyond its input partition count without needing the repartition the input stream.
   This current PR is to update the last processed offsets maintained by the OffsetManager during processing stage of the container correctly when elasticity is enabled.
   
   **Changes:**
   1. Modify OffsetManager.udpate to correctly identify the ssp even with keyBucket info is not present in the given parameter ssp
   2. Modify SystemConsumers to provide the ssp,offset to Chooser after all the registrations are done. This is because, chooser is given an ssp without keyBucket but a container could be processing multiple keyBuckets within the same ssp. So after registration for all is complete, the smallest offset is given to the chooser.
   
   **Tests:** added unit test
   
   **API changes:** no public api change
   
   **Upgrade instructions:** none
   
   **Usage instructions:** None.
   
   **Backwards compatible:** yes. does not affect the existing flow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@samza.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] rayman7718 merged pull request #1607: SAMZA-2734: [Elasticity] Update last processed offset after an envelopepe is finished processing when elasticity is enabled

[GitBox <gi...@apache.org>]

rayman7718 merged PR #1607:
URL: https://github.com/apache/samza/pull/1607


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@samza.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org