You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/08/19 00:17:20 UTC

[jira] [Commented] (METRON-381) Add support for multiple reducers in pcap_query.sh

    [ https://issues.apache.org/jira/browse/METRON-381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427395#comment-15427395 ] 

ASF GitHub Bot commented on METRON-381:
---------------------------------------

GitHub user cestella opened a pull request:

    https://github.com/apache/incubator-metron/pull/217

    METRON-381: Add support for multiple reducers in pcap_query.sh

    As a stopgap, the pcap_query.sh MapReduce job uses a single reducer to ensure that the packets captured come in the correct order. This has issues on large queries because it forces all of the data to go through one reducer. Instead, we should have a custom partitioner that will allow us to use multiple reducers.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron pcap_multi_reducer

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/217.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #217
    
----
commit 75708255120f47ba3ab47d3003e4c42f76cc99e8
Author: cstella <ce...@gmail.com>
Date:   2016-08-19T00:14:53Z

    METRON-381: Add support for multiple reducers in pcap_query.sh

----


> Add support for multiple reducers in pcap_query.sh
> --------------------------------------------------
>
>                 Key: METRON-381
>                 URL: https://issues.apache.org/jira/browse/METRON-381
>             Project: Metron
>          Issue Type: New Feature
>            Reporter: Casey Stella
>            Assignee: Casey Stella
>
> As a stopgap, the pcap_query.sh MapReduce job uses a single reducer to ensure that the packets captured come in the correct order.  This has issues on large queries because it forces all of the data to go through one reducer.  Instead, we should have a custom partitioner that will allow us to use multiple reducers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)