Use Solr / Lucene to search in a Logfile

[Robert Naczinski <ro...@googlemail.com>]

Hello,

we want to search large log4j logfiles withSolr / Lucene and find any
lines with a special argument. ( for examle: with a defined userid )
It should be a solr indput document for each row in the file. The log
file is growing continuously and the search index must be refreshed.

Has already anybidy implemented something like this and can give me a tip ?

Greetings,

Robert

Re: Use Solr / Lucene to search in a Logfile

[Matthieu Huin <ma...@wallix.com>]

Hello Robert,

At my company, we are working on a generic log collector that uses Solr 
to provide search capabilities.
What the collector does basically is this (this is greatly dumbed down !) :

* collect a log line (read it from a file, receive it from the network, 
... )
* parse it through a set of regular expressions, searching for known log 
formats ( apache CLF, ... )
* if there is a match, store the results as a set of keys/values ( url : 
http://www.apache.org , source : XXX , raw_log : xxxxxx , ... )
* insert the set as a document in the Solr backend, using the REST 
interface.

Therefore I would advise you to adapt this workflow to suit your own 
needs : have a script looking for new lines in your log file, parse them 
in order to extract the relevant information you need, store the results 
as keys/values sets, then insert them into Solr via a http call.

My company's product is probably overkill for what you need to do, and 
we'd probably need to develop a specific log parser for your log format, 
but if you are willing to give it a try feel free to contact me !

Greetings,

Matthieu HUIN

On 06/05/2011 21:40, Robert Naczinski wrote:
> Hi,
>
> thanks for the reply. I did not know that.
>
> Is there still a way to use Solr or Lucene? Or Apache Nutch would be not be bad.
>
> Could I maybe write a customized DIH?
>
> Greetings,
>
> Robert
>
> 2011/5/6 Otis Gospodnetic<ot...@yahoo.com>:
>> Loggly.com

Re: Use Solr / Lucene to search in a Logfile

[Robert Naczinski <ro...@googlemail.com>]

Hi,

thanks for the reply. I did not know that.

Is there still a way to use Solr or Lucene? Or Apache Nutch would be not be bad.

Could I maybe write a customized DIH?

Greetings,

Robert

2011/5/6 Otis Gospodnetic <ot...@yahoo.com>:
> Loggly.com

Re: Use Solr / Lucene to search in a Logfile

[Otis Gospodnetic <ot...@yahoo.com>]

Hi Robert,

Have you considered just using Loggly.com ?

Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



----- Original Message ----
> From: Robert Naczinski <ro...@googlemail.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, May 6, 2011 3:01:45 AM
> Subject: Use Solr / Lucene to search in a Logfile
> 
> Hello,
> 
> we want to search large log4j logfiles withSolr / Lucene and find  any
> lines with a special argument. ( for examle: with a defined userid  )
> It should be a solr indput document for each row in the file. The  log
> file is growing continuously and the search index must be  refreshed.
> 
> Has already anybidy implemented something like this and can  give me a tip ?
> 
> Greetings,
> 
> Robert
>