Flume sink not flushing

[Josh <jo...@mydrivesolutions.com>]

Hi,

We have a problem were events are sitting to be flushed but never seem to be, can anyone see why? I can see the events in the channel's data directory and they look to be picked up by the sink but don't get flushed.

agent.sinks.bla.type                           = hdfs
agent.sinks.bla.channel                    = bla_channel
agent.sinks.bla.hdfs.path                  = s3n://bucket-bla-reports/summary/%Y-%m-%d/
agent.sinks.bla.hdfs.fileType            = DataStream
agent.sinks.bla.hdfs.batchSize        = 5
agent.sinks.bla.hdfs.writeFormat     = Text
agent.sinks.bla.serializer                   = com.bla.flume.serialization.HeaderAndBodyTextEventSerializer$Builder
agent.sinks.bla.serializer.columns   = log_type reporting_bucket subscription_id
agent.sinks.bla.serializer.format       = BodyCSV
agent.sinks.bla.serializer.delimiter    = ,
agent.sinks.bla.serializer.appendNewline   = false
agent.sinks.bla.serializer.distanceMeasure = MILES
agent.sinks.bla.hdfs.maxOpenFiles               = 5000
agent.sinks.bla.hdfs.rollSize                           = 0
agent.sinks.bla.hdfs.rollCount                        = 0
agent.sinks.bla.hdfs.rollInterval                      = 0
agent.sinks.bla.hdfs.callTimeout                    = 60000
agent.sinks.bla.hdfs.fileSuffix                          = .csv
agent.sinks.bla.hdfs.inUseSuffix                     = .tmp
agent.sinks.bla.hdfs.filePrefix                          = Summary.%y-%m-%d.%H.%M


Thanks,
Josh Myers


Josh Myers - Linux Sys Admin
MyDrive Solutions Limited
josh.myers@mydrivesolutions.com
M: +44(0)7968 778 105
O: +44(0)1483 685550


-- 
www.mydrivesolutions.com

This email and any attachments is private and confidential. If you have 
received this message in error please remove it from your systems and 
notify the author.
MyDrive Solutions Limited is registered in England and Wales, No 07330334. 
Registered office: Surrey Technology Centre, 40 Occam Road, Guildford GU2 
7YG, UK