[jira] [Commented] (CRUNCH-680) Kafka Source should split very large partitions

["Micah Whitacre (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CRUNCH-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781860#comment-16781860 ] 

Micah Whitacre commented on CRUNCH-680:
---------------------------------------

Changes look good and the PR was merged.

> Kafka Source should split very large partitions
> -----------------------------------------------
>
>                 Key: CRUNCH-680
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-680
>             Project: Crunch
>          Issue Type: Improvement
>          Components: IO
>            Reporter: Andrew Olson
>            Assignee: Micah Whitacre
>            Priority: Minor
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> If a single Kafka partition has a very large number of messages, the map task for that partition can take a long time to run leading to potential timeout problems. We should limit the number of messages assigned to each split so that the workload is more evenly balanced.
> Based on our testing we have determined that 5 million messages should be a generally reasonable default for the maximum split size, with a configuration property (org.apache.crunch.kafka.split.max) provided to optionally override that value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)