Kafka Producer for Sending Log Data

["Tony S. Wu" <to...@gmail.com>]

Hi,

I am looking to send log file periodically to Kafka. Before I set out to
write my own producer, I was wonder what everyone uses to send log file to
Kafka through HTTP API. Ideally it should also prune the log and have some
sort of error handling.

Thanks very much.

Tony S. Wu

Re: Kafka Producer for Sending Log Data

[Ram Vittal <ra...@gmail.com>]

Hi Tony,

We are using custom logging API that wraps Kafka producer as singleton for each app. The custom API takes structured log data and converts to JSON, writes to app specific Kafka topic. Then that topic is bridged to Logstash consumer and logs get ingested to elastic search. Now these logs can be searched, filtered or visualized. 

Thanks,
Ram Vittal 

Sent from my iPhone

> On Mar 20, 2017, at 10:44 AM, Tony S. Wu <to...@gmail.com> wrote:
> 
> Hi,
> 
> I am looking to send log file periodically to Kafka. Before I set out to
> write my own producer, I was wonder what everyone uses to send log file to
> Kafka through HTTP API. Ideally it should also prune the log and have some
> sort of error handling.
> 
> Thanks very much.
> 
> Tony S. Wu