Re: Suggestions of pulling local application logs into Kafka

[Ewen Cheslack-Postava <ew...@confluent.io>]

Kafka Connect can definitely be used for this -- it's one of the reasons we
designed it with standalone mode (
http://docs.confluent.io/3.0.0/connect/userguide.html#workers). For the
specific connector, we include a very simple File connector with Kafka
which will just take each line and send it to Kafka with a simple String
schema. However, it's intentionally very limited (as an easy reference for
starting new connectors), so it doesn't do any log parsing, multi-line
handling, multiple files, tracking rolling files, etc. For that you'd need
to write a new connector. The closest one I'm familiar with is a syslog
connector, which you can find listed on connectors.confluent.io

-Ewen

On Tue, May 3, 2016 at 2:07 PM, David Birdsong <da...@gmail.com>
wrote:

> On Tue, May 3, 2016 at 1:00 PM Banias H <ba...@gmail.com> wrote:
>
> > I should add Flume is not an option for various reasons.
> >
> > On Tue, May 3, 2016 at 2:49 PM, Banias H <ba...@gmail.com> wrote:
> >
> > > We use Kafka (0.9.x) internally in our pipeline and now we would like
> to
> > > ingest application logs sitting in local file system of servers
> external
> > to
> > > the Kafka cluster.
> > >
> > > We could write a producer program running on the application servers to
> > > push files to Kafka. However we wonder if we can leverage Kafka Connect
> > to
> > > pull files into Kafka. It requires a connector to access local file
> > systems
> > > and I am not sure if I can make use of existing connectors or I need to
> > > write one.
> > >
> > > Any thoughts or reference would be helpful.
> >
>
> heka - great at reading arbitrary inputs, files being a first-class input,
> can write to kafka. next release should see use of librdkafka which should
> handle leader changeovers more smoothly than the existing go library.
>
>
>
> > >
> > > Thanks,
> > > B
> > >
> >
>



-- 
Thanks,
Ewen