[jira] [Commented] (METRON-233) Expose Kafka producer and consumer configs for parser topologies

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336556#comment-15336556 ] 

ASF GitHub Bot commented on METRON-233:
---------------------------------------

GitHub user cestella opened a pull request:

    https://github.com/apache/incubator-metron/pull/160

    METRON-233: Expose Kafka producer and consumer configs for parser topologies

    We need the ability to tune KafkaSpout and KafkaWriter properties in a more granular way.  Currently we do not expose more than a few parameters when creating the KafkaProducer in the KafkaWriter and do not expose almost any of the KafkaSpout configurations.
    
    This PR should allow for the KafkaWriter to have arbitrary KafkaProducer configs exposed in the zookeeper config and have the full range of KafkaSpout configs be exposed from the CLI used to start parser topologies.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron kafka_writer_config

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/160.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #160
    
----
commit 3eebff1f4c016b33d90fd2d0cfbacbe1125dc209
Author: cstella <ce...@gmail.com>
Date:   2016-06-17T17:47:56Z

    METRON-233: Expose Kafka producer and consumer configs for parser topologies

----


> Expose Kafka producer and consumer configs for parser topologies
> ----------------------------------------------------------------
>
>                 Key: METRON-233
>                 URL: https://issues.apache.org/jira/browse/METRON-233
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Ajay Yadav
>            Assignee: Casey Stella
>
> To tune parser topologies for different traffic and message size requires configuring kafka consumer and producer. Its important to configure properties like fetchSizeBytes, bufferSizeBytes for consumer and batch.size, buffer.memory for producer. We should expose consumer and producer configs via a property file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)