You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@samza.apache.org by dnishimura <gi...@git.apache.org> on 2018/12/13 01:11:25 UTC

[GitHub] samza pull request #860: SAMZA-1985: Startpoint and StartpointManager implem...

GitHub user dnishimura opened a pull request:

    https://github.com/apache/samza/pull/860

    SAMZA-1985: Startpoint and StartpointManager implementation.

    This is the first PR for [SEP-18](https://cwiki.apache.org/confluence/display/SAMZA/SEP-18%3A+Startpoints+-+Manipulating+Starting+Offsets+for+Input+Streams). Please see updated SEP-18 for details.
    This PR implements the StartpointManager and Startpoint model and the initial integration with the OffsetManager. The OffsetManager manages the deletion of Startpoints when the initial checkpoint commits happen per task after start-up.
    
    The immediate follow-ons to this PR are:
    1. Have the various `JobCoordinators` to re-map the Startpoints appropriately to each task by utilizing the `StartpointManager#groupStartpointsPerTask(SystemStreamPartition, SystemStreamPartitionGrouper)` method implemented in this PR. SEP-18 describes this in more detail.
    2. Add a new `SystemConsumer#register(SystemStreamPartition, Startpoint)` interface and implement the interface for each provided system (`KafkaSystemConsumer`, `EventHubSystemConsumer`, etc...) and integrate the new interface appropriately.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dnishimura/samza samza-1985-startpoint-manager

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/860.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #860
    
----
commit b8437e233f6d13c4ddbdc3cbe63d540d459611fe
Author: Daniel Nishimura <dn...@...>
Date:   2018-12-12T22:49:51Z

    SAMZA-1985: Startpoint and StartpointManager implementation.

----


---

[GitHub] samza pull request #860: SAMZA-1985: Startpoint and StartpointManager implem...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/samza/pull/860


---