You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jeff Courtade <co...@gmail.com> on 2016/12/06 18:55:31 UTC
solr audit logging
Hello,
Could someone point me in the correct direction for this?
I am being asked to setup an "audit.log" or audit trail for writes and
changes to documents.
i do not know where to begin something like this.
I am guessing it is just configuration of log4j but that is about as far as
i can go...
maybe something to do with logging things through only the update handler
or something?
Anyone bearing a cluestick is welcome.
--
Thanks,
Jeff Courtade
M: 240.507.6116
Re: solr audit logging
Posted by Jeff Courtade <co...@gmail.com>.
Thanks very much every one.
They will probably pursue custom code to see if they can get this data and
log it.
J
--
Thanks,
Jeff Courtade
M: 240.507.6116
On Tue, Dec 6, 2016 at 7:07 PM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> You know - if I had to build this, I would consider slurping up the
> relevant log entries (if they exist) and feeding them to Kafka - then your
> people who want to analyze what happened can get those entries again and
> again (Think of Kafka kind of like a persistent messaging store that can
> store log entries or anything you want...)
>
> Of course, how much work you'd have to put into that depends on the
> technical skill of whoever is going to consume this stuff...
>
> Also, a plain old relational database can easily hold these things as well
> -and the code to parse the log messages into some simple tables wouldn't be
> that difficult... There are probably existing examples / projects...
>
> Naturally - if the standard log entries do NOT get you what you need, then
> it gets to be more of an effort, although adding an extension to Solr isn't
> too hard once you understand the process...
>
> Ping back and let us know what you find in the logs and if you want more
> "advice" -- which you should always take with a grain of salt...
>
> On Tue, Dec 6, 2016 at 3:56 PM, John Bickerstaff <john@johnbickerstaff.com
> >
> wrote:
>
> > If you can identify currently-logged messages that give you what you need
> > (even if you have to modify or process them afterwards) you can easily
> make
> > a custom log4j config that grabs ONLY what you want and dumps it into a
> > separate file...
> >
> > I'm pretty sure I've seen all the request coming through in my SOLR log
> > files...
> >
> > In case that helps...
> >
> > On Tue, Dec 6, 2016 at 2:08 PM, Alexandre Rafalovitch <
> arafalov@gmail.com>
> > wrote:
> >
> >> There is also Jetty level access log which shows the requests, though
> >> it may not show the HTTP PUT bodies.
> >>
> >> Finally, various online monitoring services probably have agents that
> >> integrate with Solr to show what's happening. Usually costs money
> >> though.
> >>
> >> Regards,
> >> Alex.
> >> ----
> >> http://www.solr-start.com/ - Resources for Solr users, new and
> >> experienced
> >>
> >>
> >> On 6 December 2016 at 14:34, Jeff Courtade <co...@gmail.com>
> >> wrote:
> >> > Thanks very much the trace idea is a brilliant way to dig into it. Did
> >> not
> >> > occur to me.
> >> >
> >> > I had another coworker suggest the custom
> >> >
> >> > http://lucene.apache.org/solr/6_3_0/solr-core/org/apache/sol
> >> r/update/processor/LogUpdateProcessorFactory.html
> >> >
> >> >
> >> > this is beyond my litmited abilites.
> >> >
> >> >
> >> > I will see what we can dig up out of the logs...
> >> >
> >> >
> >> > the original request was this...
> >> >
> >> >
> >> > "Is there any configuration, plugin, or application that will create
> an
> >> > audit trail for Solr requests? We have teams that would like to be
> able
> >> to
> >> > pull back changes/requests to documents in solr given a time period.
> The
> >> > information they would like to retrieve is the request to solr, where
> it
> >> > came from, and what the request did."
> >> >
> >> >
> >> > I am starting to think there is not a simple solution to this. I was
> >> hoping
> >> > there was an UpdateAudit class or something I could flip a switch on
> or
> >> > some such...
> >> >
> >> >
> >> >
> >> > On Tue, Dec 6, 2016 at 2:20 PM, Alexandre Rafalovitch <
> >> arafalov@gmail.com>
> >> > wrote:
> >> >
> >> >> You could turn the trace mode for everything in the Admin UI (under
> >> >> logs/levels) and see if any of the existing information is sufficient
> >> >> for your needs. If yes, then you change log level in the
> configuration
> >> >> just for that class/element.
> >> >>
> >> >> Alternatively, you could do a custom UpdateRequestProcessor in the
> >> >> request handler(s) that deal with update. Or perhaps
> >> >> LogUpdateProcessor (that's in every standard chain) is sufficient:
> >> >> http://www.solr-start.com/javadoc/solr-lucene/org/
> >> >> apache/solr/update/processor/LogUpdateProcessorFactory.html
> >> >>
> >> >> But it is also possible that the audit.log is something that has a
> >> >> specific format that other tools use. So, you could start from asking
> >> >> how that file would be used and then working backwards into Solr.
> >> >> Which would most likely be a custom URP, as I mentioned earlier.
> >> >>
> >> >> Regards,
> >> >> Alex.
> >> >> P.s. Remember that there are full document updates and partial
> >> >> updates. What you want to log about that is your business level
> >> >> decision.
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks,
> >> >
> >> > Jeff Courtade
> >> > M: 240.507.6116
> >>
> >
> >
>
Re: solr audit logging
Posted by Jeff Courtade <co...@gmail.com>.
They wanted out of the box solutions.
This is what I found too that it would be custom. i was hoping i just was
not finding something obvious.
Jeff Courtade
M: 240.507.6116
On Dec 6, 2016 7:07 PM, "John Bickerstaff" <jo...@johnbickerstaff.com> wrote:
> You know - if I had to build this, I would consider slurping up the
> relevant log entries (if they exist) and feeding them to Kafka - then your
> people who want to analyze what happened can get those entries again and
> again (Think of Kafka kind of like a persistent messaging store that can
> store log entries or anything you want...)
>
> Of course, how much work you'd have to put into that depends on the
> technical skill of whoever is going to consume this stuff...
>
> Also, a plain old relational database can easily hold these things as well
> -and the code to parse the log messages into some simple tables wouldn't be
> that difficult... There are probably existing examples / projects...
>
> Naturally - if the standard log entries do NOT get you what you need, then
> it gets to be more of an effort, although adding an extension to Solr isn't
> too hard once you understand the process...
>
> Ping back and let us know what you find in the logs and if you want more
> "advice" -- which you should always take with a grain of salt...
>
> On Tue, Dec 6, 2016 at 3:56 PM, John Bickerstaff <john@johnbickerstaff.com
> >
> wrote:
>
> > If you can identify currently-logged messages that give you what you need
> > (even if you have to modify or process them afterwards) you can easily
> make
> > a custom log4j config that grabs ONLY what you want and dumps it into a
> > separate file...
> >
> > I'm pretty sure I've seen all the request coming through in my SOLR log
> > files...
> >
> > In case that helps...
> >
> > On Tue, Dec 6, 2016 at 2:08 PM, Alexandre Rafalovitch <
> arafalov@gmail.com>
> > wrote:
> >
> >> There is also Jetty level access log which shows the requests, though
> >> it may not show the HTTP PUT bodies.
> >>
> >> Finally, various online monitoring services probably have agents that
> >> integrate with Solr to show what's happening. Usually costs money
> >> though.
> >>
> >> Regards,
> >> Alex.
> >> ----
> >> http://www.solr-start.com/ - Resources for Solr users, new and
> >> experienced
> >>
> >>
> >> On 6 December 2016 at 14:34, Jeff Courtade <co...@gmail.com>
> >> wrote:
> >> > Thanks very much the trace idea is a brilliant way to dig into it. Did
> >> not
> >> > occur to me.
> >> >
> >> > I had another coworker suggest the custom
> >> >
> >> > http://lucene.apache.org/solr/6_3_0/solr-core/org/apache/sol
> >> r/update/processor/LogUpdateProcessorFactory.html
> >> >
> >> >
> >> > this is beyond my litmited abilites.
> >> >
> >> >
> >> > I will see what we can dig up out of the logs...
> >> >
> >> >
> >> > the original request was this...
> >> >
> >> >
> >> > "Is there any configuration, plugin, or application that will create
> an
> >> > audit trail for Solr requests? We have teams that would like to be
> able
> >> to
> >> > pull back changes/requests to documents in solr given a time period.
> The
> >> > information they would like to retrieve is the request to solr, where
> it
> >> > came from, and what the request did."
> >> >
> >> >
> >> > I am starting to think there is not a simple solution to this. I was
> >> hoping
> >> > there was an UpdateAudit class or something I could flip a switch on
> or
> >> > some such...
> >> >
> >> >
> >> >
> >> > On Tue, Dec 6, 2016 at 2:20 PM, Alexandre Rafalovitch <
> >> arafalov@gmail.com>
> >> > wrote:
> >> >
> >> >> You could turn the trace mode for everything in the Admin UI (under
> >> >> logs/levels) and see if any of the existing information is sufficient
> >> >> for your needs. If yes, then you change log level in the
> configuration
> >> >> just for that class/element.
> >> >>
> >> >> Alternatively, you could do a custom UpdateRequestProcessor in the
> >> >> request handler(s) that deal with update. Or perhaps
> >> >> LogUpdateProcessor (that's in every standard chain) is sufficient:
> >> >> http://www.solr-start.com/javadoc/solr-lucene/org/
> >> >> apache/solr/update/processor/LogUpdateProcessorFactory.html
> >> >>
> >> >> But it is also possible that the audit.log is something that has a
> >> >> specific format that other tools use. So, you could start from asking
> >> >> how that file would be used and then working backwards into Solr.
> >> >> Which would most likely be a custom URP, as I mentioned earlier.
> >> >>
> >> >> Regards,
> >> >> Alex.
> >> >> P.s. Remember that there are full document updates and partial
> >> >> updates. What you want to log about that is your business level
> >> >> decision.
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks,
> >> >
> >> > Jeff Courtade
> >> > M: 240.507.6116
> >>
> >
> >
>
Re: solr audit logging
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
You know - if I had to build this, I would consider slurping up the
relevant log entries (if they exist) and feeding them to Kafka - then your
people who want to analyze what happened can get those entries again and
again (Think of Kafka kind of like a persistent messaging store that can
store log entries or anything you want...)
Of course, how much work you'd have to put into that depends on the
technical skill of whoever is going to consume this stuff...
Also, a plain old relational database can easily hold these things as well
-and the code to parse the log messages into some simple tables wouldn't be
that difficult... There are probably existing examples / projects...
Naturally - if the standard log entries do NOT get you what you need, then
it gets to be more of an effort, although adding an extension to Solr isn't
too hard once you understand the process...
Ping back and let us know what you find in the logs and if you want more
"advice" -- which you should always take with a grain of salt...
On Tue, Dec 6, 2016 at 3:56 PM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> If you can identify currently-logged messages that give you what you need
> (even if you have to modify or process them afterwards) you can easily make
> a custom log4j config that grabs ONLY what you want and dumps it into a
> separate file...
>
> I'm pretty sure I've seen all the request coming through in my SOLR log
> files...
>
> In case that helps...
>
> On Tue, Dec 6, 2016 at 2:08 PM, Alexandre Rafalovitch <ar...@gmail.com>
> wrote:
>
>> There is also Jetty level access log which shows the requests, though
>> it may not show the HTTP PUT bodies.
>>
>> Finally, various online monitoring services probably have agents that
>> integrate with Solr to show what's happening. Usually costs money
>> though.
>>
>> Regards,
>> Alex.
>> ----
>> http://www.solr-start.com/ - Resources for Solr users, new and
>> experienced
>>
>>
>> On 6 December 2016 at 14:34, Jeff Courtade <co...@gmail.com>
>> wrote:
>> > Thanks very much the trace idea is a brilliant way to dig into it. Did
>> not
>> > occur to me.
>> >
>> > I had another coworker suggest the custom
>> >
>> > http://lucene.apache.org/solr/6_3_0/solr-core/org/apache/sol
>> r/update/processor/LogUpdateProcessorFactory.html
>> >
>> >
>> > this is beyond my litmited abilites.
>> >
>> >
>> > I will see what we can dig up out of the logs...
>> >
>> >
>> > the original request was this...
>> >
>> >
>> > "Is there any configuration, plugin, or application that will create an
>> > audit trail for Solr requests? We have teams that would like to be able
>> to
>> > pull back changes/requests to documents in solr given a time period. The
>> > information they would like to retrieve is the request to solr, where it
>> > came from, and what the request did."
>> >
>> >
>> > I am starting to think there is not a simple solution to this. I was
>> hoping
>> > there was an UpdateAudit class or something I could flip a switch on or
>> > some such...
>> >
>> >
>> >
>> > On Tue, Dec 6, 2016 at 2:20 PM, Alexandre Rafalovitch <
>> arafalov@gmail.com>
>> > wrote:
>> >
>> >> You could turn the trace mode for everything in the Admin UI (under
>> >> logs/levels) and see if any of the existing information is sufficient
>> >> for your needs. If yes, then you change log level in the configuration
>> >> just for that class/element.
>> >>
>> >> Alternatively, you could do a custom UpdateRequestProcessor in the
>> >> request handler(s) that deal with update. Or perhaps
>> >> LogUpdateProcessor (that's in every standard chain) is sufficient:
>> >> http://www.solr-start.com/javadoc/solr-lucene/org/
>> >> apache/solr/update/processor/LogUpdateProcessorFactory.html
>> >>
>> >> But it is also possible that the audit.log is something that has a
>> >> specific format that other tools use. So, you could start from asking
>> >> how that file would be used and then working backwards into Solr.
>> >> Which would most likely be a custom URP, as I mentioned earlier.
>> >>
>> >> Regards,
>> >> Alex.
>> >> P.s. Remember that there are full document updates and partial
>> >> updates. What you want to log about that is your business level
>> >> decision.
>> >>
>> >
>> >
>> >
>> > --
>> > Thanks,
>> >
>> > Jeff Courtade
>> > M: 240.507.6116
>>
>
>
Re: solr audit logging
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
If you can identify currently-logged messages that give you what you need
(even if you have to modify or process them afterwards) you can easily make
a custom log4j config that grabs ONLY what you want and dumps it into a
separate file...
I'm pretty sure I've seen all the request coming through in my SOLR log
files...
In case that helps...
On Tue, Dec 6, 2016 at 2:08 PM, Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> There is also Jetty level access log which shows the requests, though
> it may not show the HTTP PUT bodies.
>
> Finally, various online monitoring services probably have agents that
> integrate with Solr to show what's happening. Usually costs money
> though.
>
> Regards,
> Alex.
> ----
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 6 December 2016 at 14:34, Jeff Courtade <co...@gmail.com> wrote:
> > Thanks very much the trace idea is a brilliant way to dig into it. Did
> not
> > occur to me.
> >
> > I had another coworker suggest the custom
> >
> > http://lucene.apache.org/solr/6_3_0/solr-core/org/apache/
> solr/update/processor/LogUpdateProcessorFactory.html
> >
> >
> > this is beyond my litmited abilites.
> >
> >
> > I will see what we can dig up out of the logs...
> >
> >
> > the original request was this...
> >
> >
> > "Is there any configuration, plugin, or application that will create an
> > audit trail for Solr requests? We have teams that would like to be able
> to
> > pull back changes/requests to documents in solr given a time period. The
> > information they would like to retrieve is the request to solr, where it
> > came from, and what the request did."
> >
> >
> > I am starting to think there is not a simple solution to this. I was
> hoping
> > there was an UpdateAudit class or something I could flip a switch on or
> > some such...
> >
> >
> >
> > On Tue, Dec 6, 2016 at 2:20 PM, Alexandre Rafalovitch <
> arafalov@gmail.com>
> > wrote:
> >
> >> You could turn the trace mode for everything in the Admin UI (under
> >> logs/levels) and see if any of the existing information is sufficient
> >> for your needs. If yes, then you change log level in the configuration
> >> just for that class/element.
> >>
> >> Alternatively, you could do a custom UpdateRequestProcessor in the
> >> request handler(s) that deal with update. Or perhaps
> >> LogUpdateProcessor (that's in every standard chain) is sufficient:
> >> http://www.solr-start.com/javadoc/solr-lucene/org/
> >> apache/solr/update/processor/LogUpdateProcessorFactory.html
> >>
> >> But it is also possible that the audit.log is something that has a
> >> specific format that other tools use. So, you could start from asking
> >> how that file would be used and then working backwards into Solr.
> >> Which would most likely be a custom URP, as I mentioned earlier.
> >>
> >> Regards,
> >> Alex.
> >> P.s. Remember that there are full document updates and partial
> >> updates. What you want to log about that is your business level
> >> decision.
> >>
> >
> >
> >
> > --
> > Thanks,
> >
> > Jeff Courtade
> > M: 240.507.6116
>
Re: solr audit logging
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
There is also Jetty level access log which shows the requests, though
it may not show the HTTP PUT bodies.
Finally, various online monitoring services probably have agents that
integrate with Solr to show what's happening. Usually costs money
though.
Regards,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 6 December 2016 at 14:34, Jeff Courtade <co...@gmail.com> wrote:
> Thanks very much the trace idea is a brilliant way to dig into it. Did not
> occur to me.
>
> I had another coworker suggest the custom
>
> http://lucene.apache.org/solr/6_3_0/solr-core/org/apache/solr/update/processor/LogUpdateProcessorFactory.html
>
>
> this is beyond my litmited abilites.
>
>
> I will see what we can dig up out of the logs...
>
>
> the original request was this...
>
>
> "Is there any configuration, plugin, or application that will create an
> audit trail for Solr requests? We have teams that would like to be able to
> pull back changes/requests to documents in solr given a time period. The
> information they would like to retrieve is the request to solr, where it
> came from, and what the request did."
>
>
> I am starting to think there is not a simple solution to this. I was hoping
> there was an UpdateAudit class or something I could flip a switch on or
> some such...
>
>
>
> On Tue, Dec 6, 2016 at 2:20 PM, Alexandre Rafalovitch <ar...@gmail.com>
> wrote:
>
>> You could turn the trace mode for everything in the Admin UI (under
>> logs/levels) and see if any of the existing information is sufficient
>> for your needs. If yes, then you change log level in the configuration
>> just for that class/element.
>>
>> Alternatively, you could do a custom UpdateRequestProcessor in the
>> request handler(s) that deal with update. Or perhaps
>> LogUpdateProcessor (that's in every standard chain) is sufficient:
>> http://www.solr-start.com/javadoc/solr-lucene/org/
>> apache/solr/update/processor/LogUpdateProcessorFactory.html
>>
>> But it is also possible that the audit.log is something that has a
>> specific format that other tools use. So, you could start from asking
>> how that file would be used and then working backwards into Solr.
>> Which would most likely be a custom URP, as I mentioned earlier.
>>
>> Regards,
>> Alex.
>> P.s. Remember that there are full document updates and partial
>> updates. What you want to log about that is your business level
>> decision.
>>
>
>
>
> --
> Thanks,
>
> Jeff Courtade
> M: 240.507.6116
Re: solr audit logging
Posted by Jeff Courtade <co...@gmail.com>.
Thanks very much the trace idea is a brilliant way to dig into it. Did not
occur to me.
I had another coworker suggest the custom
http://lucene.apache.org/solr/6_3_0/solr-core/org/apache/solr/update/processor/LogUpdateProcessorFactory.html
this is beyond my litmited abilites.
I will see what we can dig up out of the logs...
the original request was this...
"Is there any configuration, plugin, or application that will create an
audit trail for Solr requests? We have teams that would like to be able to
pull back changes/requests to documents in solr given a time period. The
information they would like to retrieve is the request to solr, where it
came from, and what the request did."
I am starting to think there is not a simple solution to this. I was hoping
there was an UpdateAudit class or something I could flip a switch on or
some such...
On Tue, Dec 6, 2016 at 2:20 PM, Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> You could turn the trace mode for everything in the Admin UI (under
> logs/levels) and see if any of the existing information is sufficient
> for your needs. If yes, then you change log level in the configuration
> just for that class/element.
>
> Alternatively, you could do a custom UpdateRequestProcessor in the
> request handler(s) that deal with update. Or perhaps
> LogUpdateProcessor (that's in every standard chain) is sufficient:
> http://www.solr-start.com/javadoc/solr-lucene/org/
> apache/solr/update/processor/LogUpdateProcessorFactory.html
>
> But it is also possible that the audit.log is something that has a
> specific format that other tools use. So, you could start from asking
> how that file would be used and then working backwards into Solr.
> Which would most likely be a custom URP, as I mentioned earlier.
>
> Regards,
> Alex.
> P.s. Remember that there are full document updates and partial
> updates. What you want to log about that is your business level
> decision.
>
--
Thanks,
Jeff Courtade
M: 240.507.6116
Re: solr audit logging
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
You could turn the trace mode for everything in the Admin UI (under
logs/levels) and see if any of the existing information is sufficient
for your needs. If yes, then you change log level in the configuration
just for that class/element.
Alternatively, you could do a custom UpdateRequestProcessor in the
request handler(s) that deal with update. Or perhaps
LogUpdateProcessor (that's in every standard chain) is sufficient:
http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/update/processor/LogUpdateProcessorFactory.html
But it is also possible that the audit.log is something that has a
specific format that other tools use. So, you could start from asking
how that file would be used and then working backwards into Solr.
Which would most likely be a custom URP, as I mentioned earlier.
Regards,
Alex.
P.s. Remember that there are full document updates and partial
updates. What you want to log about that is your business level
decision.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 6 December 2016 at 13:55, Jeff Courtade <co...@gmail.com> wrote:
> Hello,
>
> Could someone point me in the correct direction for this?
>
> I am being asked to setup an "audit.log" or audit trail for writes and
> changes to documents.
>
> i do not know where to begin something like this.
>
> I am guessing it is just configuration of log4j but that is about as far as
> i can go...
>
> maybe something to do with logging things through only the update handler
> or something?
>
> Anyone bearing a cluestick is welcome.
>
> --
> Thanks,
>
> Jeff Courtade
> M: 240.507.6116