You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Milind Vaidya <ka...@gmail.com> on 2017/03/03 21:48:22 UTC
Fast way search data in kafka
I have 6 broker kafka setup.
I have retention period of 48 hrs.
To debug if certain data has reached kafka or not I am using command line
consumer to then piping to grep. But it will take huge amount of time and
may not succeed as well.
Is there an other way to search something in kafka without using consumer?
Re: Fast way search data in kafka
Posted by Milind Vaidya <ka...@gmail.com>.
Yup. I hacked a small script in bash to do it for all files and per file as
weil.
Thanks.
On Thu, Mar 23, 2017 at 2:31 PM, Marko Bonaći <ma...@sematext.com>
wrote:
> You can use something like this to get a comma-separated list of all filed
> in a folder:
>
> ls -l | awk '{print $9}' ORS=','
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Thu, Mar 23, 2017 at 9:28 PM, Milind Vaidya <ka...@gmail.com> wrote:
>
> > That looks like a faster option.
> >
> > Now the thing is --file requires list of comma separated files. Is there
> > any way to look at all files in a directory ?
> >
> >
> > I tried *log but did not work or I will have to script something to do
> that
> > ?
> >
> > On Sat, Mar 4, 2017 at 9:04 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hi Milind,
> > >
> > > You can try the DumpSegmentTool to read the logs at broker machines
> > > directly as well:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > System+Tools#SystemTools-DumpLogSegment
> > >
> > > Guozhang
> > >
> > > On Sat, Mar 4, 2017 at 9:48 AM, Anish Mashankar <
> > anish@systeminsights.com>
> > > wrote:
> > >
> > > > Try Presto https://prestodb.io. It may solve your problem.
> > > >
> > > > On Sat, 4 Mar 2017, 03:18 Milind Vaidya, <ka...@gmail.com> wrote:
> > > >
> > > > > I have 6 broker kafka setup.
> > > > >
> > > > > I have retention period of 48 hrs.
> > > > >
> > > > > To debug if certain data has reached kafka or not I am using
> command
> > > line
> > > > > consumer to then piping to grep. But it will take huge amount of
> time
> > > and
> > > > > may not succeed as well.
> > > > >
> > > > > Is there an other way to search something in kafka without using
> > > > consumer?
> > > > >
> > > > --
> > > >
> > > > Regards,
> > > > Anish Samir Mashankar
> > > > R&D Engineer
> > > > System Insights
> > > > +91-9789870733
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
Re: Fast way search data in kafka
Posted by Marko Bonaći <ma...@sematext.com>.
You can use something like this to get a comma-separated list of all filed
in a folder:
ls -l | awk '{print $9}' ORS=','
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Thu, Mar 23, 2017 at 9:28 PM, Milind Vaidya <ka...@gmail.com> wrote:
> That looks like a faster option.
>
> Now the thing is --file requires list of comma separated files. Is there
> any way to look at all files in a directory ?
>
>
> I tried *log but did not work or I will have to script something to do that
> ?
>
> On Sat, Mar 4, 2017 at 9:04 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Milind,
> >
> > You can try the DumpSegmentTool to read the logs at broker machines
> > directly as well:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > System+Tools#SystemTools-DumpLogSegment
> >
> > Guozhang
> >
> > On Sat, Mar 4, 2017 at 9:48 AM, Anish Mashankar <
> anish@systeminsights.com>
> > wrote:
> >
> > > Try Presto https://prestodb.io. It may solve your problem.
> > >
> > > On Sat, 4 Mar 2017, 03:18 Milind Vaidya, <ka...@gmail.com> wrote:
> > >
> > > > I have 6 broker kafka setup.
> > > >
> > > > I have retention period of 48 hrs.
> > > >
> > > > To debug if certain data has reached kafka or not I am using command
> > line
> > > > consumer to then piping to grep. But it will take huge amount of time
> > and
> > > > may not succeed as well.
> > > >
> > > > Is there an other way to search something in kafka without using
> > > consumer?
> > > >
> > > --
> > >
> > > Regards,
> > > Anish Samir Mashankar
> > > R&D Engineer
> > > System Insights
> > > +91-9789870733
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: Fast way search data in kafka
Posted by Milind Vaidya <ka...@gmail.com>.
That looks like a faster option.
Now the thing is --file requires list of comma separated files. Is there
any way to look at all files in a directory ?
I tried *log but did not work or I will have to script something to do that
?
On Sat, Mar 4, 2017 at 9:04 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Milind,
>
> You can try the DumpSegmentTool to read the logs at broker machines
> directly as well:
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> System+Tools#SystemTools-DumpLogSegment
>
> Guozhang
>
> On Sat, Mar 4, 2017 at 9:48 AM, Anish Mashankar <an...@systeminsights.com>
> wrote:
>
> > Try Presto https://prestodb.io. It may solve your problem.
> >
> > On Sat, 4 Mar 2017, 03:18 Milind Vaidya, <ka...@gmail.com> wrote:
> >
> > > I have 6 broker kafka setup.
> > >
> > > I have retention period of 48 hrs.
> > >
> > > To debug if certain data has reached kafka or not I am using command
> line
> > > consumer to then piping to grep. But it will take huge amount of time
> and
> > > may not succeed as well.
> > >
> > > Is there an other way to search something in kafka without using
> > consumer?
> > >
> > --
> >
> > Regards,
> > Anish Samir Mashankar
> > R&D Engineer
> > System Insights
> > +91-9789870733
> >
>
>
>
> --
> -- Guozhang
>
Re: Fast way search data in kafka
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Milind,
You can try the DumpSegmentTool to read the logs at broker machines
directly as well:
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-DumpLogSegment
Guozhang
On Sat, Mar 4, 2017 at 9:48 AM, Anish Mashankar <an...@systeminsights.com>
wrote:
> Try Presto https://prestodb.io. It may solve your problem.
>
> On Sat, 4 Mar 2017, 03:18 Milind Vaidya, <ka...@gmail.com> wrote:
>
> > I have 6 broker kafka setup.
> >
> > I have retention period of 48 hrs.
> >
> > To debug if certain data has reached kafka or not I am using command line
> > consumer to then piping to grep. But it will take huge amount of time and
> > may not succeed as well.
> >
> > Is there an other way to search something in kafka without using
> consumer?
> >
> --
>
> Regards,
> Anish Samir Mashankar
> R&D Engineer
> System Insights
> +91-9789870733
>
--
-- Guozhang
Re: Fast way search data in kafka
Posted by Anish Mashankar <an...@systeminsights.com>.
Try Presto https://prestodb.io. It may solve your problem.
On Sat, 4 Mar 2017, 03:18 Milind Vaidya, <ka...@gmail.com> wrote:
> I have 6 broker kafka setup.
>
> I have retention period of 48 hrs.
>
> To debug if certain data has reached kafka or not I am using command line
> consumer to then piping to grep. But it will take huge amount of time and
> may not succeed as well.
>
> Is there an other way to search something in kafka without using consumer?
>
--
Regards,
Anish Samir Mashankar
R&D Engineer
System Insights
+91-9789870733