You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Prateek Jain J <pr...@ericsson.com> on 2017/02/17 12:17:23 UTC
logging query received
Hi All,
How can I log queries received by solr from applications. By query, I mean full URL with parameters executed by application. For example,
I expect to see something like this
http://solr:8389/my_collection?q=*:*&fq=eventRecordTimestamp:[NOW-60000MILLISECONDS TO NOW]
I tried to enable debug logs for org.apache.solr package (java) and enable all logging levels from solr admin UI (image attached). I am hoping there should be a simple way for achieving this and something silly is what I am missing here.
[cid:image001.png@01D28917.CA388300]
Regards,
Prateek Jain
Team: Totoro
Mobile: +353 894 391716
Re: logging query received
Posted by "alessandro.benedetti" <a....@sease.io>.
If you enabled the logging for org.apache.solr.core you should be fine.
You can also go more fine grained if you don't need part of the logs.
Just remember that the UI will show only from the warning level.
If you want to see the query log you need to access the log files.
N.B. in a produciton environment I would not recommend to log at Info level
all the queries received.
It will affect the performances.
I would suggest to log only the slow queries.
[1] https://cwiki.apache.org/confluence/display/solr/Configuring+Logging
Cheers
-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
View this message in context: http://lucene.472066.n3.nabble.com/logging-query-received-tp4320937p4320943.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: logging query received
Posted by Prateek Jain J <pr...@ericsson.com>.
Thanks Alexandre,
I was on a load server so, couldn't change any code but I just enabled debug logging from Admin UI and I was able to see query in solr log file; which I turned off after 5 minutes. Thanks again for full list of options available for various scenarios.
Regards,
Prateek Jain
-----Original Message-----
From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
Sent: 17 February 2017 02:33 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: logging query received
There is actually several ways to answer this depending on the level of precision your situation requires. And, of course, there are trade-offs.
One issue to keep in mind is what you mean by "parameters". Do you want to include all the explicit defaults and overrides that the Request Handler will add itself? Or just what the calling URL has. Logging at different steps will give you different answers here. Also, some parameters can be sent in the request body, not just in the URL.
Now in terms of options:
One answer is that logging should be happening in the middleware that talks to Solr. That's where you transform UI requests into what will go to Solr and you already have the code to log it anyway you want. But that's harder if you have multiple clients all talking to Solr or if you have trust in your smart-firewall and expose Solr directly to the web.
The next option is to enable standard HTTP access log on the bundled Jetty level. However, Jetty is an implementation detail, so you should keep that in mind as you migrate.
Next, I am pretty sure there is a debug setting that logs all the query elements. You can just enable TRACE on the highest Solr level on the test system and then pick the specific message you like most. Then, enable that in log4j.config file for your production. That would be low level reporting, possibly in a format you need to post-process. Depending on where that debug is coming from, this may or may not include the defaults/overrides.
You could enable echoParams=all or echoParams=explicit, which will then send the request parameters with the reply in the JSON format. You'd need to extract it from that somehow though. Again, it would be easy if you had a single contact point in middleware as well.
Finally, you could do a custom search component and - since it has access to the request parameter map - log it any way you want. Or a custom UpdateRequestProcessor, I think they can access that info as well.
Or, if you can't touch anything, but your network/cpu is not overloaded, you could run Wireshark and capture the HTTP requests right off the network traffic. That's bringing gun with a silencer to a knife fight, but perhaps you would enjoy the bragging rights. Or - more realistically - perhaps you want to capture POST body content, response timing, encoding abnormalities or other debug-level issues.
Hope this helps,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 17 February 2017 at 07:17, Prateek Jain J <pr...@ericsson.com>
wrote:
>
>
> Hi All,
>
>
>
> How can I log queries received by solr from applications. By query, I
> mean full URL with parameters executed by application. For example,
>
>
>
> I expect to see something like this
>
>
>
> http://solr:8389/my_collection?q=*:*&fq=eventRecordTimestamp:[NOW-
> 60000MILLISECONDS TO NOW]
>
>
>
> I tried to enable debug logs for org.apache.solr package (java) and
> enable all logging levels from solr admin UI (image attached). I am
> hoping there should be a simple way for achieving this and something
> silly is what I am missing here.
>
>
>
>
>
>
>
> Regards,
>
> Prateek Jain
>
> Team: Totoro
>
> Mobile: +353 894 391716
>
>
>
Re: logging query received
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
There is actually several ways to answer this depending on the level of
precision your situation requires. And, of course, there are trade-offs.
One issue to keep in mind is what you mean by "parameters". Do you want to
include all the explicit defaults and overrides that the Request Handler
will add itself? Or just what the calling URL has. Logging at different
steps will give you different answers here. Also, some parameters can be
sent in the request body, not just in the URL.
Now in terms of options:
One answer is that logging should be happening in the middleware that talks
to Solr. That's where you transform UI requests into what will go to Solr
and you already have the code to log it anyway you want. But that's harder
if you have multiple clients all talking to Solr or if you have trust in
your smart-firewall and expose Solr directly to the web.
The next option is to enable standard HTTP access log on the bundled Jetty
level. However, Jetty is an implementation detail, so you should keep that
in mind as you migrate.
Next, I am pretty sure there is a debug setting that logs all the query
elements. You can just enable TRACE on the highest Solr level on the test
system and then pick the specific message you like most. Then, enable that
in log4j.config file for your production. That would be low level
reporting, possibly in a format you need to post-process. Depending on
where that debug is coming from, this may or may not include the
defaults/overrides.
You could enable echoParams=all or echoParams=explicit, which will then
send the request parameters with the reply in the JSON format. You'd need
to extract it from that somehow though. Again, it would be easy if you had
a single contact point in middleware as well.
Finally, you could do a custom search component and - since it has access
to the request parameter map - log it any way you want. Or a custom
UpdateRequestProcessor, I think they can access that info as well.
Or, if you can't touch anything, but your network/cpu is not overloaded,
you could run Wireshark and capture the HTTP requests right off the network
traffic. That's bringing gun with a silencer to a knife fight, but perhaps
you would enjoy the bragging rights. Or - more realistically - perhaps you
want to capture POST body content, response timing, encoding abnormalities
or other debug-level issues.
Hope this helps,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 17 February 2017 at 07:17, Prateek Jain J <pr...@ericsson.com>
wrote:
>
>
> Hi All,
>
>
>
> How can I log queries received by solr from applications. By query, I mean
> full URL with parameters executed by application. For example,
>
>
>
> I expect to see something like this
>
>
>
> http://solr:8389/my_collection?q=*:*&fq=eventRecordTimestamp:[NOW-
> 60000MILLISECONDS TO NOW]
>
>
>
> I tried to enable debug logs for org.apache.solr package (java) and enable
> all logging levels from solr admin UI (image attached). I am hoping there
> should be a simple way for achieving this and something silly is what I am
> missing here.
>
>
>
>
>
>
>
> Regards,
>
> Prateek Jain
>
> Team: Totoro
>
> Mobile: +353 894 391716
>
>
>