You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@samza.apache.org by vjagadish1989 <gi...@git.apache.org> on 2017/11/01 20:01:20 UTC

[GitHub] samza pull request #348: SAMZA-1479 : Refactor KafkaCheckpointManager, Kafka...

GitHub user vjagadish1989 opened a pull request:

    https://github.com/apache/samza/pull/348

    SAMZA-1479 : Refactor KafkaCheckpointManager, KafkaCheckpointLogKey and their tests

    Notable changes:
    * Rewrite `KafkaCheckpointLogKey` into two classes - an immutable class, and a SerDe
    * Remove dependency on static setters in the `KafkaCheckpointLogKey`
    * Change lifecycle of components in KafkaCheckpointManager 
      - It's safe to start producers and consumers during `start` as opposed to lazy loading them during writes, and reads.
      - Initialize systemProducer and systemConsumer during construction
    * Simplify logic for ignoring checkpoint validations
    * Re-write checkpointManager#readLog() to use a simpler API.  
    * Remove unnecessary complexity after the migration from 0.8
    * Remove unnecessary locking in startup, and shut-down
    * Remove dependencies on SimpleConsumer configs like bufferSize, fetchSize, socketTimeout
    * Refactor KafkaCheckpointManagerFactory and remove static getCheckpointSystemNameAndFactory
    * Bug-fix : Register the taskName correctly (instead of using a dummy string for the taskName)
    
    * Add unit tests to verify more checkpoint scenarios
    * Consolidate unit tests into utils for creating producer, consumer and admin instances
    * Convert/consolidate most long-running integration tests into unit tests


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vjagadish1989/samza kafka-checkpointmanager-refactor

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/348.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #348
    
----
commit c8b225f1b44fc180a5d6bb385e917a42bc54d97f
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T01:37:09Z

    Refactor checkpoint log key

commit f28d2e24a1d91cb5cefad6d5d0245c694d6d83ac
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T01:43:12Z

    Merge branch 'master' of https://github.com/apache/samza

commit 289eaa58730965a51d4b32e788568f554d645eae
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T02:24:02Z

    Fixes for KafkaCheckpointManager

commit 9e533283ff991dd227f81c0af363ff5a2c2580e0
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T03:11:54Z

    Fixes for KafkaCheckpointManager

commit 9ffcf5391f3e78cb1e402bba8f8a2abdc4a4a729
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T05:13:09Z

    checkpoint manager refactor

commit c10159d2cca756e65661307c7d9fa1f8ce7857ff
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T06:01:42Z

    rewrite kcm

commit 7db01525084b2c91410323531c8716abcbfd4e77
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T06:03:59Z

    Renamed LogKey class

commit 8db287198d9239729478d1631e6b535865123b34
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-20T06:33:16Z

    Initial pass at getting tests to compile

commit fc7d30b2ecd03c5e10f3c9c8d2482b2e6a922a44
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-21T02:11:10Z

    Fixed checkpoint serdes, and a bunch of unit tests

commit 904446173f920eefc676c561f7780768ef84b886
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-26T01:36:36Z

    stage commits for KCM

commit 4f04fe204555759a59e89f9e07d4f9c63e8dd6bf
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-31T15:38:54Z

    Cleanup tests, separate unit tests and integration tests

commit 34208f71c62b59e0bb060305f35c36c630248bdd
Author: Jagadish <jv...@linkedin.com>
Date:   2017-10-31T17:01:16Z

    Merge with latest master

commit d5411c7d4ac793dd880b9b1550f29a8a153cace6
Author: Jagadish <jv...@linkedin.com>
Date:   2017-11-01T19:17:10Z

    Rewrite KafkaCheckpointManagerFactory. Refactor static methods. Remove duplicate getCheckpointTopicProperties

----


---

[GitHub] samza pull request #348: SAMZA-1479 : Refactor KafkaCheckpointManager, Kafka...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/samza/pull/348


---