You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Saurabh Sharma <sa...@nviz.com> on 2016/01/25 13:10:00 UTC

filename not appearing in kafka topic event header

Hi,

I am trying to configure SpoolingDirectory as a source and Kafka as a sink in apache flume with following configuration.
Could you please help me to understand how I can send fileheader in an event header to Kafka topic. Am I missing anything in following configuration.


agent.sources = spoolDir
agent.channels = memoryChannel
agent.sinks = sink
agent.sources.spoolDir.interceptors = i1

#Source configuration
agent.sources.spoolDir.type = spooldir
agent.sources.spoolDir.spoolDir = D:/CommerceSense/spoolDir
agent.sources.spoolDir.fileHeader = true
agent.sources.spoolDir.basenameHeader = true
agent.sources.spoolDir.deserializer = LINE
agent.sources.spoolDir.interceptors.i1.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
agent.sources.spoolDir.interceptors.i1.preserveExisting = true
agent.sources.spoolDir.interceptors.i1.prefix = test
agent.sources.spoolDir.channels = memoryChannel


#Channel Configuration
agent.channels.memoryChannel.type = memory


#Sink Configuration
agent.sinks.sink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.sink.topic = cdnLogsTopic
agent.sinks.sink.brokerList = localhost:9092
agent.sinks.sink.batchSize = 100
agent.sinks.sink.channel = memoryChannel