[jira] [Created] (FLUME-2446) Flume -> ElasticSearch loosing data

["Tigran (JIRA)" <ji...@apache.org>]

Tigran created FLUME-2446:
-----------------------------

             Summary: Flume -> ElasticSearch loosing data
                 Key: FLUME-2446
                 URL: https://issues.apache.org/jira/browse/FLUME-2446
             Project: Flume
          Issue Type: Bug
         Environment: Ubuntu 14.04.1 LTS

            Reporter: Tigran


I use flume to put events into elasticsearch, but not all my events are put there. 
This is metrics json file:
{"SOURCE.avroSource":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"0","AppendBatchReceivedCount":"0","EventAcceptedCount":"21","AppendReceivedCount":"21","StopTime":"0","StartTime":"1408553130069","EventReceivedCount":"21","AppendAcceptedCount":"21"},"SINK.esSyslogSink":{"BatchCompleteCount":"0","ConnectionFailedCount":"0","EventDrainAttemptCount":"0","ConnectionCreatedCount":"1","Type":"SINK","BatchEmptyCount":"69","ConnectionClosedCount":"0","EventDrainSuccessCount":"0","StopTime":"0","StartTime":"1408553129240","BatchUnderflowCount":"0"},"CHANNEL.avroMemoryChannel":{"EventPutSuccessCount":"21","ChannelFillPercentage":"0.0","Type":"CHANNEL","StopTime":"0","EventPutAttemptCount":"21","ChannelSize":"0","StartTime":"1408553129232","EventTakeSuccessCount":"21","ChannelCapacity":"1000","EventTakeAttemptCount":"162"},"SINK.esAvroSink":{"BatchCompleteCount":"2","ConnectionFailedCount":"0","EventDrainAttemptCount":"14","ConnectionCreatedCount":"1","Type":"SINK","BatchEmptyCount":"68","ConnectionClosedCount":"0","EventDrainSuccessCount":"14","StopTime":"0","StartTime":"1408553129245","BatchUnderflowCount":"1"},"CHANNEL.syslogMemoryChannel":{"EventPutSuccessCount":"0","ChannelFillPercentage":"0.0","Type":"CHANNEL","StopTime":"0","EventPutAttemptCount":"0","ChannelSize":"0","StartTime":"1408553129232","EventTakeSuccessCount":"0","ChannelCapacity":"1000","EventTakeAttemptCount":"69"}}

So I put 21 events, and only 14 were passed into elasticsearch (EventDrainSuccessCount":"14"). Flume do receive all 21 events: "EventAcceptedCount":"21", and also my written interceptor (that log every incoming message) shows me this. Ration sent/received is random: sometime it gets all messages, but usually it is between 6-14 of 21. 

Do somebody experienced this? No errors or warnings in the log.



--
This message was sent by Atlassian JIRA
(v6.2#6252)