[jira] [Commented] (METRON-569) Enrichment topology duplicates messages

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670820#comment-15670820 ] 

ASF GitHub Bot commented on METRON-569:
---------------------------------------

Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/359
  
    Also, @DomenicPuzio would you mind renaming this PR to METRON-569: Change acking to prevent duplicate tuples in enrichment topology so that the PR comments get replicated properly?


> Enrichment topology duplicates messages
> ---------------------------------------
>
>                 Key: METRON-569
>                 URL: https://issues.apache.org/jira/browse/METRON-569
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Domenic Puzio
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> When running the 'enrichment' topology, I get duplicate message being indexed. For example, I put 100 messages into the 'enrichment' Kafka queue and I get 175 messages onto the 'indexing' Kafka queue. This happens when I am running the 'enrichment' topology with one or more enrichment bolt.
> This is an acking issue within the JoinBolt class. When a message does not "complete" the join (like when it is the first message in a pair of message to get joined) it does not get acked. This means that this message will get replayed through Storm, causing message duplication further down the road and tons of additional overhead. Adding the correct acking resolves this problem.
> I will add the PR for this shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)