[jira] [Commented] (KAFKA-3914) Global discovery of state stores

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355415#comment-15355415 ] 

ASF GitHub Bot commented on KAFKA-3914:
---------------------------------------

GitHub user dguy opened a pull request:

    https://github.com/apache/kafka/pull/1570

    KAFKA-3914: Global discovery of state stores

    @guozhangwang @enothereska please take a look. A few things that need to be clarified
    
    1. I've added `StreamsConfig.USER_ENDPOINT_CONFIG`, but should we have separate configs for host and port or is this one config ok?
    2. `HostState` in the KIP has a `byte[]` field - not sure why and what it would be populated with
    3. The API calls in the KIP all return `Map<HostState, Set<TaskMetadata>>`, however i don't see why the  `Set` is required so I've changed it to `Map<HostState, TaskMetadata>`

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dguy/kafka kafka-3914

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/1570.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1570
    
----
commit f22ca0e680063d42240d8ceebe140ecd9e3fab48
Author: Damian Guy <da...@gmail.com>
Date:   2016-06-29T07:40:11Z

    Connect stateStores to source topics.
    Add user endpoint config to StreamsConfig.
    Collect Map<HostState, Set<TopicPartition>> in StreamPartitionAssignor
    API for retrieving Map<HostState, TaskMetadata>

----


> Global discovery of state stores
> --------------------------------
>
>                 Key: KAFKA-3914
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3914
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: streams
>    Affects Versions: 0.10.1.0
>            Reporter: Damian Guy
>            Assignee: Damian Guy
>             Fix For: 0.10.1.0
>
>
> - Update group membership data to include discovery endpoints
> - Enable discovery
> We need to attach some host and port information to {{org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfo}}
> This can then be used in {{org.apache.kafka.streams.processor.internals.StreamPartitionAssignor}} to build a {{Map<HostState, Set<TopicPartition>>}} that should be added to
> {{org.apache.kafka.streams.processor.internals.assignment.AssignmentInfo}} during the {{assign}} call.
> When {{StreamPartitionAssignor.onAssignment}} is called we should hold on to a copy of the {{Map<HostState, Set<TopicPartition>>}} for use by the discovery methods
> To enable discovery, implement the below methods on {{KafkaStreams}}
> {code}
> /**
>  * @return metadata about all tasks
>  */
> Map<HostState, Set<TaskMetadata>> getAllTasks();
> /**
>  * @param storeName requested store name
>  * @return metadata about all tasks that include
>  * storeName in this KStreams instance
>  */
> Map<HostState, Set<TaskMetadata>> getAllTasksWithStore(String storeName);
> /**
>  * @param key requested key
>  * @param storeName requested store name
>  * @return metadata about all tasks that include
>  * storeName and key in this KStreams instance
>  */
> <K> Map<HostState, Set<TaskMetadata>> getAllTasksWithKey(String storeName, K key);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)