[jira] [Closed] (SAMZA-1479) Proposal for Kafka checkpoint manager improvements

["Jagadish (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jagadish closed SAMZA-1479.
---------------------------

> Proposal for Kafka checkpoint manager improvements
> --------------------------------------------------
>
>                 Key: SAMZA-1479
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1479
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Jagadish
>            Assignee: Jagadish
>            Priority: Trivial
>
> This proposal adds the following improvements to KafkaCheckpointManager for better testability:
> *  Rewrite `KafkaCheckpointLogKey` into two classes - an immutable class, and a SerDe:
> * Remove dependency on static setters in the `KafkaCheckpointLogKey`
> * Change lifecycle of components in KafkaCheckpointManager 
>   -- It's safe to start producers and consumers during `start` as opposed to lazy loading them during writes, and reads.
>   -- Initialize systemProducer and systemConsumer during construction
> * Simplify logic for ignoring checkpoint validations
> * Re-write checkpointManager#readLog() to use a simpler API.  
> * Remove unnecessary complexity after the migration from 0.8
> * Remove unnecessary locking in startup, and shut-down
> * Remove dependencies on SimpleConsumer configs like bufferSize, fetchSize, socketTimeout
> * Refactor KafkaCheckpointManagerFactory and remove static getCheckpointSystemNameAndFactory
> * Bug-fix : Register the taskName correctly (instead of using a dummy string for the taskName)
> Testing improvements:
> * Add unit tests to verify more checkpoint scenarios
> * Consolidate unit tests into utils for creating producer, consumer and admin instances
> * Convert/consolidate most long-running integration tests into unit tests



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)