[jira] [Created] (METRON-569) Enrichment topology duplicates messages

["Domenic Puzio (JIRA)" <ji...@apache.org>]

Domenic Puzio created METRON-569:
------------------------------------

             Summary: Enrichment topology duplicates messages
                 Key: METRON-569
                 URL: https://issues.apache.org/jira/browse/METRON-569
             Project: Metron
          Issue Type: Bug
            Reporter: Domenic Puzio


When running the 'enrichment' topology, I get duplicate message being indexed. For example, I put 100 messages into the 'enrichment' Kafka queue and I get 175 messages onto the 'indexing' Kafka queue. This happens when I am running the 'enrichment' topology with one or more enrichment bolt.

This is an acking issue within the JoinBolt class. When a message does not "complete" the join (like when it is the first message in a pair of message to get joined) it does not get acked. This means that this message will get replayed through Storm, causing message duplication further down the road and tons of additional overhead. Adding the correct acking resolves this problem.

I will add the PR for this shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)