[GitHub] [samza] shanthoosh opened a new pull request #954: SAMZA-2129: Move the offset comparison check to choose the lowest offset to SystemConsumers.

[GitBox <gi...@apache.org>]

shanthoosh opened a new pull request #954: SAMZA-2129: Move the offset comparison check to choose the lowest offset to SystemConsumers.
URL: https://github.com/apache/samza/pull/954
 
 
   This patch is comprised of the following changes:
   
   * Moves the offset comparison check to SystemConsumers
    
        * Currently the offset comparator check to find a lowest for a `SystemStreamPartition` is duplicated in some implementations of  SystemConsumer API.
        * Some `SystemConsumer` implementations do not perform this offset comparator check
   in the implementation of register method.
        * Moving this one level up from `SystemConsumer.register(SystemStreamPartition, offset)` API implementation to  `SystemConsumers.register(SystemStreamPartition, offset)` API implementation removes unnecessary duplication and ensures functional correctness.
   
   * Fixes to the test-cases in `TestSystemConsumer`.
   
   There should be no functional breakages introduced by moving the offset-comparator check to `SystemConsumers` layer. Here's why. Metadata and I/O topics currently use SystemConsumer API to read data. 
   * Checkpoint topic: This is a stream with one partition and is log-compacted. Samza-container reads from the beginning of this stream. There's no offset comparisons required for this topic.
   * ChangeLog topic: To read change-log topic-partition, SamzaContainer uses `StorageManagerUtil.getStartingOffset` to read from the change-log topic and the offset comparator check to choose the lowest offset is currently performed within it.
   * Coordinator topic: This is a stream with one partition and is log-compacted. SamzaContainer/ApplicationMaster is currently reads from the beginning of this stream to get the entire state.
   *  I/O topics: `SamzaContainer` currently reads  data from input and output topics through `SystemConsumers` API. Moving this offset comparator check should not affect the existing functionality.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services