You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Anil <an...@gmail.com> on 2016/06/05 06:08:28 UTC

Indexing logs in Solr

Hi ,

i would like to index logs using to enable search on it in our application.

The problem would be index and stored size as log files size would go upto
terabytes.

is there any way to use highlight feature without storing ?

i found following link where Benedetti Alessandro mentioned about custom
highlighter on url field.

http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html

Any ideas would be helpful. Thanks.

Cheers,
Anil

Re: Indexing logs in Solr

Posted by Anil <an...@gmail.com>.

Any external application wont be integrated with our application.

are there any custom solr highlighter ? Thanks.

On 6 June 2016 at 04:12, Joe Lawson <jl...@opensourceconnections.com>
wrote:

> Flume and Logstash can both ship to Solr.
> On Jun 5, 2016 2:11 PM, "Otis Gospodnetic" <ot...@gmail.com>
> wrote:
>
> > You can ship SOLR logs to Logsene or any other log management service and
> > not worry too much about their storage/size.
> >
> > Otis
> >
> > > On Jun 5, 2016, at 02:08, Anil <an...@gmail.com> wrote:
> > >
> > > Hi ,
> > >
> > > i would like to index logs using to enable search on it in our
> > application.
> > >
> > > The problem would be index and stored size as log files size would go
> > upto
> > > terabytes.
> > >
> > > is there any way to use highlight feature without storing ?
> > >
> > > i found following link where Benedetti Alessandro mentioned about
> custom
> > > highlighter on url field.
> > >
> > >
> >
> http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html
> > >
> > > Any ideas would be helpful. Thanks.
> > >
> > > Cheers,
> > > Anil
> >
>

Re: Indexing logs in Solr

Posted by Joe Lawson <jl...@opensourceconnections.com>.

Flume and Logstash can both ship to Solr.
On Jun 5, 2016 2:11 PM, "Otis Gospodnetic" <ot...@gmail.com>
wrote:

> You can ship SOLR logs to Logsene or any other log management service and
> not worry too much about their storage/size.
>
> Otis
>
> > On Jun 5, 2016, at 02:08, Anil <an...@gmail.com> wrote:
> >
> > Hi ,
> >
> > i would like to index logs using to enable search on it in our
> application.
> >
> > The problem would be index and stored size as log files size would go
> upto
> > terabytes.
> >
> > is there any way to use highlight feature without storing ?
> >
> > i found following link where Benedetti Alessandro mentioned about custom
> > highlighter on url field.
> >
> >
> http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html
> >
> > Any ideas would be helpful. Thanks.
> >
> > Cheers,
> > Anil
>

Re: Indexing logs in Solr

Posted by Otis Gospodnetic <ot...@gmail.com>.

You can ship SOLR logs to Logsene or any other log management service and not worry too much about their storage/size.

Otis

> On Jun 5, 2016, at 02:08, Anil <an...@gmail.com> wrote:
> 
> Hi ,
> 
> i would like to index logs using to enable search on it in our application.
> 
> The problem would be index and stored size as log files size would go upto
> terabytes.
> 
> is there any way to use highlight feature without storing ?
> 
> i found following link where Benedetti Alessandro mentioned about custom
> highlighter on url field.
> 
> http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html
> 
> Any ideas would be helpful. Thanks.
> 
> Cheers,
> Anil

Re: Indexing logs in Solr

Posted by Anil <an...@gmail.com>.

Hi Eric and Benedetti Alessandro*,*

do you have any inputs on the solution given the following link ?
http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html

Thanks,
Anil

On 5 June 2016 at 11:56, Anil <an...@gmail.com> wrote:

> Thanks IIan. I will look into this.
> In our case, logs are attached to some application information and its
> linked to other product information.
>
> Based on the log information, user will be navigated to other features of
> product. So we cannot directly decouple log search from our application.
>
> Thanks,
> Anil
>
> On 5 June 2016 at 11:42, Ilan Schwarts <il...@gmail.com> wrote:
>
>> How about using "logstash" for this? I know its ES and not solr, but it is
>> a free tool that is out there and no need to re-invent the wheel
>> On Jun 5, 2016 9:09 AM, "Anil" <an...@gmail.com> wrote:
>>
>> > Hi ,
>> >
>> > i would like to index logs using to enable search on it in our
>> application.
>> >
>> > The problem would be index and stored size as log files size would go
>> upto
>> > terabytes.
>> >
>> > is there any way to use highlight feature without storing ?
>> >
>> > i found following link where Benedetti Alessandro mentioned about custom
>> > highlighter on url field.
>> >
>> >
>> >
>> http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html
>> >
>> > Any ideas would be helpful. Thanks.
>> >
>> > Cheers,
>> > Anil
>> >
>>
>
>

Re: Indexing logs in Solr

Posted by SIDDHAST® Roshan <ro...@siddhast.com>.

hi anil,

as i know storing is required for highlighting and KWIC in solr.
one way you can do it is do not store logs in solr and retrieve them
directly from file or DB and highlight via regular expression

Roshan


On 6/5/16, Anil <an...@gmail.com> wrote:
> Thanks IIan. I will look into this.
> In our case, logs are attached to some application information and its
> linked to other product information.
>
> Based on the log information, user will be navigated to other features of
> product. So we cannot directly decouple log search from our application.
>
> Thanks,
> Anil
>
> On 5 June 2016 at 11:42, Ilan Schwarts <il...@gmail.com> wrote:
>
>> How about using "logstash" for this? I know its ES and not solr, but it
>> is
>> a free tool that is out there and no need to re-invent the wheel
>> On Jun 5, 2016 9:09 AM, "Anil" <an...@gmail.com> wrote:
>>
>> > Hi ,
>> >
>> > i would like to index logs using to enable search on it in our
>> application.
>> >
>> > The problem would be index and stored size as log files size would go
>> upto
>> > terabytes.
>> >
>> > is there any way to use highlight feature without storing ?
>> >
>> > i found following link where Benedetti Alessandro mentioned about
>> > custom
>> > highlighter on url field.
>> >
>> >
>> >
>> http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html
>> >
>> > Any ideas would be helpful. Thanks.
>> >
>> > Cheers,
>> > Anil
>> >
>>
>


-- 

Roshan Agarwal
Director sales
Siddhast® Ip innovation (P) ltd
907 chandra vihar colony
Jhansi-284002
M:+917376314900

Re: Indexing logs in Solr

Posted by Anil <an...@gmail.com>.

Thanks IIan. I will look into this.
In our case, logs are attached to some application information and its
linked to other product information.

Based on the log information, user will be navigated to other features of
product. So we cannot directly decouple log search from our application.

Thanks,
Anil

On 5 June 2016 at 11:42, Ilan Schwarts <il...@gmail.com> wrote:

> How about using "logstash" for this? I know its ES and not solr, but it is
> a free tool that is out there and no need to re-invent the wheel
> On Jun 5, 2016 9:09 AM, "Anil" <an...@gmail.com> wrote:
>
> > Hi ,
> >
> > i would like to index logs using to enable search on it in our
> application.
> >
> > The problem would be index and stored size as log files size would go
> upto
> > terabytes.
> >
> > is there any way to use highlight feature without storing ?
> >
> > i found following link where Benedetti Alessandro mentioned about custom
> > highlighter on url field.
> >
> >
> >
> http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html
> >
> > Any ideas would be helpful. Thanks.
> >
> > Cheers,
> > Anil
> >
>

Re: Indexing logs in Solr

Posted by Ilan Schwarts <il...@gmail.com>.

How about using "logstash" for this? I know its ES and not solr, but it is
a free tool that is out there and no need to re-invent the wheel
On Jun 5, 2016 9:09 AM, "Anil" <an...@gmail.com> wrote:

> Hi ,
>
> i would like to index logs using to enable search on it in our application.
>
> The problem would be index and stored size as log files size would go upto
> terabytes.
>
> is there any way to use highlight feature without storing ?
>
> i found following link where Benedetti Alessandro mentioned about custom
> highlighter on url field.
>
>
> http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html
>
> Any ideas would be helpful. Thanks.
>
> Cheers,
> Anil
>