You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Eva Sheeva <sh...@gmail.com> on 2019/09/23 14:57:04 UTC
Shipping Kafka logs
Hello
Can you please point me to the doc or guide me on shipping kafka logs to
elk. I understand it is a little different than how we ship other logs to
elk. Kindly help me with this
Thanks
Re: Shipping Kafka logs
Posted by Eva Sheeva <sh...@gmail.com>.
Thanks Liam, I'm trying with Filebeat but i'm unsure what filter should i
use for kafka logs. I need all the controller.log, server.log
and state-change.log to be shipped. I tried using grok filter but that is
throwing error. Do you have any info on the filter to be used for Kafka
logs?
Regards
On Tue, Sep 24, 2019 at 1:11 PM Liam Clarke <li...@adscale.co.nz>
wrote:
> Cc list
>
> ---------- Forwarded message ---------
> From: Liam Clarke <li...@adscale.co.nz>
> Date: Tue, 24 Sep. 2019, 7:40 pm
> Subject: Re: Shipping Kafka logs
> To: Eva Sheeva <sh...@gmail.com>
>
>
> Ah, for that you need something like Filebeat - it's an agent that forwards
> monitored log files to ELK.
>
> On Tue, 24 Sep. 2019, 7:17 pm Eva Sheeva, <sh...@gmail.com> wrote:
>
> > Thank you Jacob and Liam!
> >
> > I 'm looking for the steps on shipping Kafka application logs. I want the
> > controller.log, state-change.log and server.log to be shipped to elk. Can
> > you pls help!
> >
> > Regards!
> >
> > On Tue, Sep 24, 2019 at 8:08 AM Botuck, Jacob (STL) <
> > JBotuck@express-scripts.com> wrote:
> >
> >> You can also try
> >>
> https://docs.confluent.io/current/connect/kafka-connect-elasticsearch/index.html
> >> I think both links are about shipping kafka messages to elk, not actual
> >> kafka application logs
> >>
> >> -----Original Message-----
> >> From: Liam Clarke <li...@adscale.co.nz>
> >> Sent: Monday, September 23, 2019 5:21 PM
> >> To: users@kafka.apache.org
> >> Subject: [EXTERNAL] Re: Shipping Kafka logs
> >>
> >> Hi Eva,
> >>
> >> Hope this helps.
> >> <company email system killed this link>
> >> Kind regards,
> >>
> >> Liam Clarke
> >>
> >> On Tue, Sep 24, 2019 at 2:57 AM Eva Sheeva <sh...@gmail.com>
> wrote:
> >>
> >> > Hello
> >> >
> >> > Can you please point me to the doc or guide me on shipping kafka logs
> >> > to elk. I understand it is a little different than how we ship other
> >> > logs to elk. Kindly help me with this
> >> >
> >> > Thanks
> >> >
> >>
> >
>
Fwd: Shipping Kafka logs
Posted by Liam Clarke <li...@adscale.co.nz>.
Cc list
---------- Forwarded message ---------
From: Liam Clarke <li...@adscale.co.nz>
Date: Tue, 24 Sep. 2019, 7:40 pm
Subject: Re: Shipping Kafka logs
To: Eva Sheeva <sh...@gmail.com>
Ah, for that you need something like Filebeat - it's an agent that forwards
monitored log files to ELK.
On Tue, 24 Sep. 2019, 7:17 pm Eva Sheeva, <sh...@gmail.com> wrote:
> Thank you Jacob and Liam!
>
> I 'm looking for the steps on shipping Kafka application logs. I want the
> controller.log, state-change.log and server.log to be shipped to elk. Can
> you pls help!
>
> Regards!
>
> On Tue, Sep 24, 2019 at 8:08 AM Botuck, Jacob (STL) <
> JBotuck@express-scripts.com> wrote:
>
>> You can also try
>> https://docs.confluent.io/current/connect/kafka-connect-elasticsearch/index.html
>> I think both links are about shipping kafka messages to elk, not actual
>> kafka application logs
>>
>> -----Original Message-----
>> From: Liam Clarke <li...@adscale.co.nz>
>> Sent: Monday, September 23, 2019 5:21 PM
>> To: users@kafka.apache.org
>> Subject: [EXTERNAL] Re: Shipping Kafka logs
>>
>> Hi Eva,
>>
>> Hope this helps.
>> <company email system killed this link>
>> Kind regards,
>>
>> Liam Clarke
>>
>> On Tue, Sep 24, 2019 at 2:57 AM Eva Sheeva <sh...@gmail.com> wrote:
>>
>> > Hello
>> >
>> > Can you please point me to the doc or guide me on shipping kafka logs
>> > to elk. I understand it is a little different than how we ship other
>> > logs to elk. Kindly help me with this
>> >
>> > Thanks
>> >
>>
>
Re: Shipping Kafka logs
Posted by Eva Sheeva <sh...@gmail.com>.
Thank you Jacob and Liam!
I 'm looking for the steps on shipping Kafka application logs. I want the
controller.log, state-change.log and server.log to be shipped to elk. Can
you pls help!
Regards!
On Tue, Sep 24, 2019 at 8:08 AM Botuck, Jacob (STL) <
JBotuck@express-scripts.com> wrote:
> You can also try
> https://docs.confluent.io/current/connect/kafka-connect-elasticsearch/index.html
> I think both links are about shipping kafka messages to elk, not actual
> kafka application logs
>
> -----Original Message-----
> From: Liam Clarke <li...@adscale.co.nz>
> Sent: Monday, September 23, 2019 5:21 PM
> To: users@kafka.apache.org
> Subject: [EXTERNAL] Re: Shipping Kafka logs
>
> Hi Eva,
>
> Hope this helps.
> <company email system killed this link>
> Kind regards,
>
> Liam Clarke
>
> On Tue, Sep 24, 2019 at 2:57 AM Eva Sheeva <sh...@gmail.com> wrote:
>
> > Hello
> >
> > Can you please point me to the doc or guide me on shipping kafka logs
> > to elk. I understand it is a little different than how we ship other
> > logs to elk. Kindly help me with this
> >
> > Thanks
> >
>
Shipping Kafka logs
Posted by "Botuck, Jacob (STL)" <JB...@express-scripts.com>.
You can also try https://docs.confluent.io/current/connect/kafka-connect-elasticsearch/index.html I think both links are about shipping kafka messages to elk, not actual kafka application logs
-----Original Message-----
From: Liam Clarke <li...@adscale.co.nz>
Sent: Monday, September 23, 2019 5:21 PM
To: users@kafka.apache.org
Subject: [EXTERNAL] Re: Shipping Kafka logs
Hi Eva,
Hope this helps.
<company email system killed this link>
Kind regards,
Liam Clarke
On Tue, Sep 24, 2019 at 2:57 AM Eva Sheeva <sh...@gmail.com> wrote:
> Hello
>
> Can you please point me to the doc or guide me on shipping kafka logs
> to elk. I understand it is a little different than how we ship other
> logs to elk. Kindly help me with this
>
> Thanks
>
Re: Shipping Kafka logs
Posted by Liam Clarke <li...@adscale.co.nz>.
Hi Eva,
Hope this helps.
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
Kind regards,
Liam Clarke
On Tue, Sep 24, 2019 at 2:57 AM Eva Sheeva <sh...@gmail.com> wrote:
> Hello
>
> Can you please point me to the doc or guide me on shipping kafka logs to
> elk. I understand it is a little different than how we ship other logs to
> elk. Kindly help me with this
>
> Thanks
>