[jira] [Commented] (FLINK-8516) FlinkKinesisConsumer does not balance shards over subtasks

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349100#comment-16349100 ] 

ASF GitHub Bot commented on FLINK-8516:
---------------------------------------

GitHub user tweise reopened a pull request:

    https://github.com/apache/flink/pull/5393

    [FLINK-8516] Allow for custom hash function for shard to subtask mapping in Kinesis consumer

    ## What is the purpose of the change
    
    Allow the user to customize Kinesis shard to subtask assignment in the Kinesis consumer.
    
    ## Brief change log
    
    Added pluggable shard assigner.
    
    ## Verifying this change
    
    Added unit test. 
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
    Javadoc

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tweise/flink FLINK-8516.shardHashing

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5393
    
----
commit ad4dbe6fb5bf2af52726c54b6361089ef3f4e369
Author: Thomas Weise <th...@...>
Date:   2018-01-31T01:44:44Z

    [FLINK-8516] Allow for custom hash function for shard to subtask mapping in Kinesis consumer

----


> FlinkKinesisConsumer does not balance shards over subtasks
> ----------------------------------------------------------
>
>                 Key: FLINK-8516
>                 URL: https://issues.apache.org/jira/browse/FLINK-8516
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.4.0, 1.3.2, 1.5.0
>            Reporter: Thomas Weise
>            Assignee: Thomas Weise
>            Priority: Major
>
> The hash code of the shard is used to distribute discovered shards over subtasks round robin. This works as long as shard identifiers are sequential. After shards are rebalanced in Kinesis, that may no longer be the case and the distribution become skewed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)