You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Sam Lee <sa...@yahoo.com.INVALID> on 2022/02/21 08:01:43 UTC
Storing logs in Apache Solr
Is Apache Solr suitable for use as a log aggregation and near-real-time
log querying tool?
I am thinking of using Apache Solr to store syslog messages as they
arrive. For example, by using rsyslog with its "pipe output module"
(ompipe) that pipes logs into a Python/Ruby/Shell script that then
inserts the data into Apache Solr.
Is Apache Solr suitable for this use case? I mean, if Elasticsearch can
do this, then so can Apache Solr, right? On the web, I didn't find
much information about using Apache Solr as a log aggregation and
querying tool. Most people seem to use the ELK stack that has arguably
become non-open-source in 2021.
Re: Storing logs in Apache Solr
Posted by Eric Pugh <ep...@opensourceconnections.com>.
While this is still a bit “roll your own”, you may be interested in https://solr.apache.org/guide/8_11/logs.html
> On Feb 21, 2022, at 3:54 AM, Z0ltrix <z0...@pm.me.INVALID> wrote:
>
> Hi Sam Lee,
>
> yes solr is suitable for this. We aggregate various logfiles from many hosts with minifi and send them line by line to solr. Ingestion and indexing is fine and you can query the logfiles just moments after ingestion.
>
> Regards,
> Christian
>
> ------- Original Message -------
>
> Sam Lee <sa...@yahoo.com.INVALID> schrieb am Montag, 21. Februar 2022 um 09:01:
>
>> Is Apache Solr suitable for use as a log aggregation and near-real-time
>>
>
>> log querying tool?
>>
>
>> I am thinking of using Apache Solr to store syslog messages as they
>>
>
>> arrive. For example, by using rsyslog with its "pipe output module"
>>
>
>> (ompipe) that pipes logs into a Python/Ruby/Shell script that then
>>
>
>> inserts the data into Apache Solr.
>>
>
>> Is Apache Solr suitable for this use case? I mean, if Elasticsearch can
>>
>
>> do this, then so can Apache Solr, right? On the web, I didn't find
>>
>
>> much information about using Apache Solr as a log aggregation and
>>
>
>> querying tool. Most people seem to use the ELK stack that has arguably
>>
>
>> become non-open-source in 2021.
> <publickey - z0ltrix@pm.me - 0xF0E154C5.asc>
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: Storing logs in Apache Solr
Posted by Dave <ha...@gmail.com>.
Solrs stats functions are great when analyzing logs if they are pre processed.
> On Feb 21, 2022, at 4:26 PM, Joel Bernstein <jo...@gmail.com> wrote:
>
> We use Solr for logs analytics. This is a lot more power in Solr's math
> expressions than in Elastic's aggregations and Solr also has new root cause
> analysis and event correlation query. Here are some links:
>
> https://solr.apache.org/guide/8_11/math-expressions.html
> https://solr.apache.org/guide/8_11/logs.html
> http://joelsolr.blogspot.com/2021/02/driving-down-cloud-storage-costs-with.html
> https://solr.apache.org/guide/8_11/graph.html#temporal-graph-expressions
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
>> On Mon, Feb 21, 2022 at 10:52 AM Z0ltrix <z0...@pm.me.invalid> wrote:
>>
>> No, we use a more general approach to collect all kinds of logfiles
>>
>> TailFile -> ExtractGrok -> JoltTransformJSON (make some transformation) ->
>> PutSolrRecord
>>
>>
>> And we dont use Nifi directly, we use Minifi running on each machine. So
>> we can collect all kinds of logfiles from all kinds of applications.
>>
>> ------- Original Message -------
>>
>> Sam Lee <sa...@yahoo.com.INVALID> schrieb am Montag, 21. Februar 2022
>> um 16:35:
>>
>>>> yes solr is suitable for this. We aggregate various logfiles from many
>>>
>>
>>>> hosts with minifi and send them line by line to solr. Ingestion and
>>>>
>>
>>>> indexing is fine and you can query the logfiles just moments after
>>>>
>>
>>>> ingestion.
>>>
>>
>>> Thank you for the idea. I have no experience with Apache NiFi at the
>>>
>>
>>> moment. Is your Apache NiFi set up something like this?
>>>
>>
>>>
>> https://bryanbende.com/development/2015/05/17/collecting-logs-with-apache-nifi
>>>
>>
>>> (ListenUDP -> MergeContent -> PutSolrContentStream).
Re: Storing logs in Apache Solr
Posted by Joel Bernstein <jo...@gmail.com>.
We use Solr for logs analytics. This is a lot more power in Solr's math
expressions than in Elastic's aggregations and Solr also has new root cause
analysis and event correlation query. Here are some links:
https://solr.apache.org/guide/8_11/math-expressions.html
https://solr.apache.org/guide/8_11/logs.html
http://joelsolr.blogspot.com/2021/02/driving-down-cloud-storage-costs-with.html
https://solr.apache.org/guide/8_11/graph.html#temporal-graph-expressions
Joel Bernstein
http://joelsolr.blogspot.com/
On Mon, Feb 21, 2022 at 10:52 AM Z0ltrix <z0...@pm.me.invalid> wrote:
> No, we use a more general approach to collect all kinds of logfiles
>
> TailFile -> ExtractGrok -> JoltTransformJSON (make some transformation) ->
> PutSolrRecord
>
>
> And we dont use Nifi directly, we use Minifi running on each machine. So
> we can collect all kinds of logfiles from all kinds of applications.
>
> ------- Original Message -------
>
> Sam Lee <sa...@yahoo.com.INVALID> schrieb am Montag, 21. Februar 2022
> um 16:35:
>
> > > yes solr is suitable for this. We aggregate various logfiles from many
> >
>
> > > hosts with minifi and send them line by line to solr. Ingestion and
> > >
>
> > > indexing is fine and you can query the logfiles just moments after
> > >
>
> > > ingestion.
> >
>
> > Thank you for the idea. I have no experience with Apache NiFi at the
> >
>
> > moment. Is your Apache NiFi set up something like this?
> >
>
> >
> https://bryanbende.com/development/2015/05/17/collecting-logs-with-apache-nifi
> >
>
> > (ListenUDP -> MergeContent -> PutSolrContentStream).
Re: Storing logs in Apache Solr
Posted by Z0ltrix <z0...@pm.me.INVALID>.
No, we use a more general approach to collect all kinds of logfiles
TailFile -> ExtractGrok -> JoltTransformJSON (make some transformation) -> PutSolrRecord
And we dont use Nifi directly, we use Minifi running on each machine. So we can collect all kinds of logfiles from all kinds of applications.
------- Original Message -------
Sam Lee <sa...@yahoo.com.INVALID> schrieb am Montag, 21. Februar 2022 um 16:35:
> > yes solr is suitable for this. We aggregate various logfiles from many
>
> > hosts with minifi and send them line by line to solr. Ingestion and
> >
> > indexing is fine and you can query the logfiles just moments after
> >
> > ingestion.
>
> Thank you for the idea. I have no experience with Apache NiFi at the
>
> moment. Is your Apache NiFi set up something like this?
>
> https://bryanbende.com/development/2015/05/17/collecting-logs-with-apache-nifi
>
> (ListenUDP -> MergeContent -> PutSolrContentStream).
Re: Storing logs in Apache Solr
Posted by Sam Lee <sa...@yahoo.com.INVALID>.
> yes solr is suitable for this. We aggregate various logfiles from many
> hosts with minifi and send them line by line to solr. Ingestion and
> indexing is fine and you can query the logfiles just moments after
> ingestion.
Thank you for the idea. I have no experience with Apache NiFi at the
moment. Is your Apache NiFi set up something like this?
https://bryanbende.com/development/2015/05/17/collecting-logs-with-apache-nifi
(ListenUDP -> MergeContent -> PutSolrContentStream).
Re: Storing logs in Apache Solr
Posted by Z0ltrix <z0...@pm.me.INVALID>.
Hi Sam Lee,
yes solr is suitable for this. We aggregate various logfiles from many hosts with minifi and send them line by line to solr. Ingestion and indexing is fine and you can query the logfiles just moments after ingestion.
Regards,
Christian
------- Original Message -------
Sam Lee <sa...@yahoo.com.INVALID> schrieb am Montag, 21. Februar 2022 um 09:01:
> Is Apache Solr suitable for use as a log aggregation and near-real-time
>
> log querying tool?
>
> I am thinking of using Apache Solr to store syslog messages as they
>
> arrive. For example, by using rsyslog with its "pipe output module"
>
> (ompipe) that pipes logs into a Python/Ruby/Shell script that then
>
> inserts the data into Apache Solr.
>
> Is Apache Solr suitable for this use case? I mean, if Elasticsearch can
>
> do this, then so can Apache Solr, right? On the web, I didn't find
>
> much information about using Apache Solr as a log aggregation and
>
> querying tool. Most people seem to use the ELK stack that has arguably
>
> become non-open-source in 2021.