[jira] [Updated] (SAMZA-2129) Move the offset comparison check to choose the lowest offset to SystemConsumers.

["Shanthoosh Venkataraman (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shanthoosh Venkataraman updated SAMZA-2129:
-------------------------------------------
    Description: 
Moves the offset comparison check to SystemConsumers
 * Currently the offset comparator check to find a lowest for a `SystemStreamPartition` is duplicated in some implementations of SystemConsumer API.
 * Some `SystemConsumer` implementations do not perform this offset comparator check
 in the implementation of register method.
 * Moving this one level up from `SystemConsumer.register(SystemStreamPartition, offset)` API implementation to `SystemConsumers.register(SystemStreamPartition, offset)` API implementation removes unnecessary duplication and ensures functional correctness.

There should be no functional breakages introduced by moving the offset-comparator check to `SystemConsumers` layer. Here's why. Metadata and I/O topics currently use SystemConsumer API to read data.
 * Checkpoint topic: This is a stream with one partition and is log-compacted. Samza-container reads from the beginning of this stream. There's no offset comparisons required for this topic.
 * ChangeLog topic: To read change-log topic-partition, SamzaContainer uses `StorageManagerUtil.getStartingOffset` to read from the change-log topic and the offset comparator check to choose the lowest offset is currently performed within it.
 * Coordinator topic: This is a stream with one partition and is log-compacted. SamzaContainer/ApplicationMaster is currently reads from the beginning of this stream to get the entire state.
 * I/O topics: `SamzaContainer` currently reads data from input and output topics through `SystemConsumers` API. Moving this offset comparator check should not affect the existing functionality.

  was:
* Moves the offset comparison check to SystemConsumers
 
 * Currently the offset comparator check to find a lowest for a `SystemStreamPartition` is duplicated in some implementations of SystemConsumer API.
 * Some `SystemConsumer` implementations do not perform this offset comparator check
in the implementation of register method.
 * Moving this one level up from `SystemConsumer.register(SystemStreamPartition, offset)` API implementation to `SystemConsumers.register(SystemStreamPartition, offset)` API implementation removes unnecessary duplication and ensures functional correctness.

There should be no functional breakages introduced by moving the offset-comparator check to `SystemConsumers` layer. Here's why. Metadata and I/O topics currently use SystemConsumer API to read data. 
* Checkpoint topic: This is a stream with one partition and is log-compacted. Samza-container reads from the beginning of this stream. There's no offset comparisons required for this topic.
* ChangeLog topic: To read change-log topic-partition, SamzaContainer uses `StorageManagerUtil.getStartingOffset` to read from the change-log topic and the offset comparator check to choose the lowest offset is currently performed within it.
* Coordinator topic: This is a stream with one partition and is log-compacted. SamzaContainer/ApplicationMaster is currently reads from the beginning of this stream to get the entire state.
* I/O topics: `SamzaContainer` currently reads data from input and output topics through `SystemConsumers` API. Moving this offset comparator check should not affect the existing functionality.


> Move the offset comparison check to choose the lowest offset to SystemConsumers.
> --------------------------------------------------------------------------------
>
>                 Key: SAMZA-2129
>                 URL: https://issues.apache.org/jira/browse/SAMZA-2129
>             Project: Samza
>          Issue Type: Improvement
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>            Priority: Major
>
> Moves the offset comparison check to SystemConsumers
>  * Currently the offset comparator check to find a lowest for a `SystemStreamPartition` is duplicated in some implementations of SystemConsumer API.
>  * Some `SystemConsumer` implementations do not perform this offset comparator check
>  in the implementation of register method.
>  * Moving this one level up from `SystemConsumer.register(SystemStreamPartition, offset)` API implementation to `SystemConsumers.register(SystemStreamPartition, offset)` API implementation removes unnecessary duplication and ensures functional correctness.
> There should be no functional breakages introduced by moving the offset-comparator check to `SystemConsumers` layer. Here's why. Metadata and I/O topics currently use SystemConsumer API to read data.
>  * Checkpoint topic: This is a stream with one partition and is log-compacted. Samza-container reads from the beginning of this stream. There's no offset comparisons required for this topic.
>  * ChangeLog topic: To read change-log topic-partition, SamzaContainer uses `StorageManagerUtil.getStartingOffset` to read from the change-log topic and the offset comparator check to choose the lowest offset is currently performed within it.
>  * Coordinator topic: This is a stream with one partition and is log-compacted. SamzaContainer/ApplicationMaster is currently reads from the beginning of this stream to get the entire state.
>  * I/O topics: `SamzaContainer` currently reads data from input and output topics through `SystemConsumers` API. Moving this offset comparator check should not affect the existing functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)