You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ian Holsman <li...@holsman.net> on 2007/05/10 06:43:22 UTC
Re: Requests per second/minute monitor?
Walter Underwood wrote:
> This is for monitoring -- what happened in the last 30 seconds.
> Log file analysis doesn't really do that.
>
I would respectfully disagree.
Log file analysis of each request can give you that, and a whole lot more.
you could either grab the stats via a regular cron job, or create a separate
filter to parse them real time.
It would then let you grab more sophisticated stats if you choose to.
What I would like to know is (and excuse the newbieness of the question) how
to enable solr to log a file with the following data.
- time spent (ms) in the request.
- IP# of the incoming request
- what the request was (and what handler executed it)
- a status code to signal if the request failed for some reasons
- number of rows fetched
and
- the number of rows actually returned
is this possible? (I'm using tomcat if that changes the answer).
regards
Ian
--
View this message in context: http://www.nabble.com/Re%3A-Requests-per-second-minute-monitor--tf3659369.html#a10407072
Sent from the Solr - User mailing list archive at Nabble.com.
Re: PriceJunkie.com using solr!
Posted by James liu <li...@gmail.com>.
aha,,,,it is wonderful.
2007/5/24, Mike Austin <ma...@gmail.com>:
>
> Just one.
>
> -----Original Message-----
> From: James liu [mailto:liuping.james@gmail.com]
> Sent: Wednesday, May 16, 2007 10:30 PM
> To: solr-user@lucene.apache.org
> Subject: Re: PriceJunkie.com using solr!
>
>
> how many solr instance?
>
>
> 2007/5/17, Yonik Seeley <yo...@apache.org>:
> >
> > Congrats, very nice job!
> > It's fast too.
> >
> > -Yonik
> >
> > On 5/16/07, Mike Austin <ma...@gmail.com> wrote:
> > > I just wanted to say thanks to everyone for the creation of
> solr. I've
> > been
> > > using it for a while now and I have recently brought one of my side
> > projects
> > > online. I have several other projects that will be using solr for
> it's
> > > search and facets.
> > >
> > > Please check out www.pricejunkie.com and let us know what you think..
> > You
> > > can give feedback and/or sign up on the mailing list for future
> updates.
> > > The site is very basic right now and many new and useful features plus
> > > merchants and product categories will be coming soon! I thought it
> > would be
> > > a good idea to at least have a few people use it to get some feedback
> > early
> > > and often.
> > >
> > > Some of the nice things behind the scenes that we did with solr:
> > > - created custom request handlers that have category to facet to
> > attribute
> > > caching built in
> > > - category to facet management
> > > - ability to manage facet groups (attributes within a set
> facet)
> > and assign
> > > them to categories
> > > - ability to create any category structure and share facet
> > groups
> > >
> > > - facet inheritance for any category (a facet group can be defined on
> a
> > > parent category and pushed down to all children)
> > > - ability to create sub-categories as facets instead of normal sub
> > > categories
> > > - simple xml configuration for the final outputted category
> > configuration
> > > file
> > >
> > >
> > > I'm sure there are more cool things but that is all for now. Join the
> > > mailing list to see more improvements in the future.
> > >
> > > Also.. how do I get added to the Using Solr wiki page?
> > >
> > >
> > > Thanks,
> > > Mike Austin
> >
>
>
>
> --
> regards
> jl
>
>
--
regards
jl
RE: PriceJunkie.com using solr!
Posted by Mike Austin <ma...@gmail.com>.
Just one.
-----Original Message-----
From: James liu [mailto:liuping.james@gmail.com]
Sent: Wednesday, May 16, 2007 10:30 PM
To: solr-user@lucene.apache.org
Subject: Re: PriceJunkie.com using solr!
how many solr instance?
2007/5/17, Yonik Seeley <yo...@apache.org>:
>
> Congrats, very nice job!
> It's fast too.
>
> -Yonik
>
> On 5/16/07, Mike Austin <ma...@gmail.com> wrote:
> > I just wanted to say thanks to everyone for the creation of solr. I've
> been
> > using it for a while now and I have recently brought one of my side
> projects
> > online. I have several other projects that will be using solr for it's
> > search and facets.
> >
> > Please check out www.pricejunkie.com and let us know what you think..
> You
> > can give feedback and/or sign up on the mailing list for future updates.
> > The site is very basic right now and many new and useful features plus
> > merchants and product categories will be coming soon! I thought it
> would be
> > a good idea to at least have a few people use it to get some feedback
> early
> > and often.
> >
> > Some of the nice things behind the scenes that we did with solr:
> > - created custom request handlers that have category to facet to
> attribute
> > caching built in
> > - category to facet management
> > - ability to manage facet groups (attributes within a set facet)
> and assign
> > them to categories
> > - ability to create any category structure and share facet
> groups
> >
> > - facet inheritance for any category (a facet group can be defined on a
> > parent category and pushed down to all children)
> > - ability to create sub-categories as facets instead of normal sub
> > categories
> > - simple xml configuration for the final outputted category
> configuration
> > file
> >
> >
> > I'm sure there are more cool things but that is all for now. Join the
> > mailing list to see more improvements in the future.
> >
> > Also.. how do I get added to the Using Solr wiki page?
> >
> >
> > Thanks,
> > Mike Austin
>
--
regards
jl
Re: PriceJunkie.com using solr!
Posted by James liu <li...@gmail.com>.
how many solr instance?
2007/5/17, Yonik Seeley <yo...@apache.org>:
>
> Congrats, very nice job!
> It's fast too.
>
> -Yonik
>
> On 5/16/07, Mike Austin <ma...@gmail.com> wrote:
> > I just wanted to say thanks to everyone for the creation of solr. I've
> been
> > using it for a while now and I have recently brought one of my side
> projects
> > online. I have several other projects that will be using solr for it's
> > search and facets.
> >
> > Please check out www.pricejunkie.com and let us know what you think..
> You
> > can give feedback and/or sign up on the mailing list for future updates.
> > The site is very basic right now and many new and useful features plus
> > merchants and product categories will be coming soon! I thought it
> would be
> > a good idea to at least have a few people use it to get some feedback
> early
> > and often.
> >
> > Some of the nice things behind the scenes that we did with solr:
> > - created custom request handlers that have category to facet to
> attribute
> > caching built in
> > - category to facet management
> > - ability to manage facet groups (attributes within a set facet)
> and assign
> > them to categories
> > - ability to create any category structure and share facet
> groups
> >
> > - facet inheritance for any category (a facet group can be defined on a
> > parent category and pushed down to all children)
> > - ability to create sub-categories as facets instead of normal sub
> > categories
> > - simple xml configuration for the final outputted category
> configuration
> > file
> >
> >
> > I'm sure there are more cool things but that is all for now. Join the
> > mailing list to see more improvements in the future.
> >
> > Also.. how do I get added to the Using Solr wiki page?
> >
> >
> > Thanks,
> > Mike Austin
>
--
regards
jl
Re: PriceJunkie.com using solr!
Posted by Yonik Seeley <yo...@apache.org>.
Congrats, very nice job!
It's fast too.
-Yonik
On 5/16/07, Mike Austin <ma...@gmail.com> wrote:
> I just wanted to say thanks to everyone for the creation of solr. I've been
> using it for a while now and I have recently brought one of my side projects
> online. I have several other projects that will be using solr for it's
> search and facets.
>
> Please check out www.pricejunkie.com and let us know what you think.. You
> can give feedback and/or sign up on the mailing list for future updates.
> The site is very basic right now and many new and useful features plus
> merchants and product categories will be coming soon! I thought it would be
> a good idea to at least have a few people use it to get some feedback early
> and often.
>
> Some of the nice things behind the scenes that we did with solr:
> - created custom request handlers that have category to facet to attribute
> caching built in
> - category to facet management
> - ability to manage facet groups (attributes within a set facet) and assign
> them to categories
> - ability to create any category structure and share facet groups
>
> - facet inheritance for any category (a facet group can be defined on a
> parent category and pushed down to all children)
> - ability to create sub-categories as facets instead of normal sub
> categories
> - simple xml configuration for the final outputted category configuration
> file
>
>
> I'm sure there are more cool things but that is all for now. Join the
> mailing list to see more improvements in the future.
>
> Also.. how do I get added to the Using Solr wiki page?
>
>
> Thanks,
> Mike Austin
RE: PriceJunkie.com using solr!
Posted by Mike Austin <ma...@gmail.com>.
Thanks Tim. Yes, the results are transformed with xslt in the .net page
after getting the response xml back from the handler. So, SOLR is called
via http from the very light .net page. Besides the transform, SOLR does
all of the work with the help of some of my custom classes that utilizes a
generated xml config file that has category to facet to query item type
mappings.
-----Original Message-----
From: Tim Archambault [mailto:joetomgo@gmail.com]
Sent: Thursday, May 17, 2007 11:25 AM
To: solr-user@lucene.apache.org; maustin75@gmail.com
Subject: Re: PriceJunkie.com using solr!
I did a search and noticed pages were executed through aspx. Are you using
.net to parse the xml results from SOLR? Nice site, just trying to figure
out where SOLR fits into this.
On 5/16/07, Mike Austin <ma...@gmail.com> wrote:
I just wanted to say thanks to everyone for the creation of solr. I've
been
using it for a while now and I have recently brought one of my side
projects
online. I have several other projects that will be using solr for it's
search and facets.
Please check out www.pricejunkie.com and let us know what you think..
You
can give feedback and/or sign up on the mailing list for future updates.
The site is very basic right now and many new and useful features plus
merchants and product categories will be coming soon! I thought it
would be
a good idea to at least have a few people use it to get some feedback
early
and often.
Some of the nice things behind the scenes that we did with solr:
- created custom request handlers that have category to facet to
attribute
caching built in
- category to facet management
- ability to manage facet groups (attributes within a set facet)
and assign
them to categories
- ability to create any category structure and share facet
groups
- facet inheritance for any category (a facet group can be defined on a
parent category and pushed down to all children)
- ability to create sub-categories as facets instead of normal sub
categories
- simple xml configuration for the final outputted category
configuration
file
I'm sure there are more cool things but that is all for now. Join the
mailing list to see more improvements in the future.
Also.. how do I get added to the Using Solr wiki page?
Thanks,
Mike Austin
Re: PriceJunkie.com using solr!
Posted by Tim Archambault <jo...@gmail.com>.
I did a search and noticed pages were executed through aspx. Are you using
.net to parse the xml results from SOLR? Nice site, just trying to figure
out where SOLR fits into this.
On 5/16/07, Mike Austin <ma...@gmail.com> wrote:
>
>
> I just wanted to say thanks to everyone for the creation of solr. I've
> been
> using it for a while now and I have recently brought one of my side
> projects
> online. I have several other projects that will be using solr for it's
> search and facets.
>
> Please check out www.pricejunkie.com and let us know what you think.. You
> can give feedback and/or sign up on the mailing list for future updates.
> The site is very basic right now and many new and useful features plus
> merchants and product categories will be coming soon! I thought it would
> be
> a good idea to at least have a few people use it to get some feedback
> early
> and often.
>
> Some of the nice things behind the scenes that we did with solr:
> - created custom request handlers that have category to facet to attribute
> caching built in
> - category to facet management
> - ability to manage facet groups (attributes within a set facet)
> and assign
> them to categories
> - ability to create any category structure and share facet groups
>
> - facet inheritance for any category (a facet group can be defined on a
> parent category and pushed down to all children)
> - ability to create sub-categories as facets instead of normal sub
> categories
> - simple xml configuration for the final outputted category configuration
> file
>
>
> I'm sure there are more cool things but that is all for now. Join the
> mailing list to see more improvements in the future.
>
> Also.. how do I get added to the Using Solr wiki page?
>
>
> Thanks,
> Mike Austin
>
>
Re: PriceJunkie.com using solr!
Posted by Chris Hostetter <ho...@fucit.org>.
: Also.. how do I get added to the Using Solr wiki page?
Anyone can update the wiki, just click the Login link and create an
account -- the edit controls will start showing up.
-Hoss
Re: PriceJunkie.com using solr!
Posted by Matthew Runo <mr...@zappos.com>.
Wow.
Those sound like some really great features, ones that I'd also love
to use!
What do we need to do to convince you to put them in the SOLR repo?
+--------------------------------------------------------+
| Matthew Runo
| Zappos Development
| mruno@zappos.com
| 702-943-7833
+--------------------------------------------------------+
On May 17, 2007, at 8:20 AM, Martin Grotzke wrote:
> Very nice and really fast, congrats!
>
> Are you willing to provide the mentioned features to solr users?
> I think espacially the category to facet management (facet groups)
> is really useful...
> It would be very nice to have this problem solved once... :)
>
> Cheers,
> Martin
>
>
> On Wed, 2007-05-16 at 16:28 -0500, Mike Austin wrote:
>> I just wanted to say thanks to everyone for the creation of solr.
>> I've been
>> using it for a while now and I have recently brought one of my
>> side projects
>> online. I have several other projects that will be using solr for
>> it's
>> search and facets.
>>
>> Please check out www.pricejunkie.com and let us know what you
>> think.. You
>> can give feedback and/or sign up on the mailing list for future
>> updates.
>> The site is very basic right now and many new and useful features
>> plus
>> merchants and product categories will be coming soon! I thought
>> it would be
>> a good idea to at least have a few people use it to get some
>> feedback early
>> and often.
>>
>> Some of the nice things behind the scenes that we did with solr:
>> - created custom request handlers that have category to facet to
>> attribute
>> caching built in
>> - category to facet management
>> - ability to manage facet groups (attributes within a set facet)
>> and assign
>> them to categories
>> - ability to create any category structure and share facet groups
>>
>> - facet inheritance for any category (a facet group can be defined
>> on a
>> parent category and pushed down to all children)
>> - ability to create sub-categories as facets instead of normal sub
>> categories
>> - simple xml configuration for the final outputted category
>> configuration
>> file
>>
>>
>> I'm sure there are more cool things but that is all for now. Join
>> the
>> mailing list to see more improvements in the future.
>>
>> Also.. how do I get added to the Using Solr wiki page?
>>
>>
>> Thanks,
>> Mike Austin
>>
> --
> Martin Grotzke
> http://www.javakaffee.de/blog/
Re: PriceJunkie.com using solr!
Posted by Martin Grotzke <ma...@javakaffee.de>.
Very nice and really fast, congrats!
Are you willing to provide the mentioned features to solr users?
I think espacially the category to facet management (facet groups)
is really useful...
It would be very nice to have this problem solved once... :)
Cheers,
Martin
On Wed, 2007-05-16 at 16:28 -0500, Mike Austin wrote:
> I just wanted to say thanks to everyone for the creation of solr. I've been
> using it for a while now and I have recently brought one of my side projects
> online. I have several other projects that will be using solr for it's
> search and facets.
>
> Please check out www.pricejunkie.com and let us know what you think.. You
> can give feedback and/or sign up on the mailing list for future updates.
> The site is very basic right now and many new and useful features plus
> merchants and product categories will be coming soon! I thought it would be
> a good idea to at least have a few people use it to get some feedback early
> and often.
>
> Some of the nice things behind the scenes that we did with solr:
> - created custom request handlers that have category to facet to attribute
> caching built in
> - category to facet management
> - ability to manage facet groups (attributes within a set facet) and assign
> them to categories
> - ability to create any category structure and share facet groups
>
> - facet inheritance for any category (a facet group can be defined on a
> parent category and pushed down to all children)
> - ability to create sub-categories as facets instead of normal sub
> categories
> - simple xml configuration for the final outputted category configuration
> file
>
>
> I'm sure there are more cool things but that is all for now. Join the
> mailing list to see more improvements in the future.
>
> Also.. how do I get added to the Using Solr wiki page?
>
>
> Thanks,
> Mike Austin
>
--
Martin Grotzke
http://www.javakaffee.de/blog/
PriceJunkie.com using solr!
Posted by Mike Austin <ma...@gmail.com>.
I just wanted to say thanks to everyone for the creation of solr. I've been
using it for a while now and I have recently brought one of my side projects
online. I have several other projects that will be using solr for it's
search and facets.
Please check out www.pricejunkie.com and let us know what you think.. You
can give feedback and/or sign up on the mailing list for future updates.
The site is very basic right now and many new and useful features plus
merchants and product categories will be coming soon! I thought it would be
a good idea to at least have a few people use it to get some feedback early
and often.
Some of the nice things behind the scenes that we did with solr:
- created custom request handlers that have category to facet to attribute
caching built in
- category to facet management
- ability to manage facet groups (attributes within a set facet) and assign
them to categories
- ability to create any category structure and share facet groups
- facet inheritance for any category (a facet group can be defined on a
parent category and pushed down to all children)
- ability to create sub-categories as facets instead of normal sub
categories
- simple xml configuration for the final outputted category configuration
file
I'm sure there are more cool things but that is all for now. Join the
mailing list to see more improvements in the future.
Also.. how do I get added to the Using Solr wiki page?
Thanks,
Mike Austin
PriceJunkie.com using solr!
Posted by Mike Austin <ma...@gmail.com>.
I just wanted to say thanks to everyone for the creation of solr. I've been
using it for a while now and I have recently brought one of my side projects
online. I have several other projects that will be using solr for it's
search and facets.
Please check out www.pricejunkie.com and let us know what you think.. You
can give feedback and/or sign up on the mailing list for future updates.
The site is very basic right now and many new and useful features plus
merchants and product categories will be coming soon! I thought it would be
a good idea to at least have a few people use it to get some feedback early
and often.
Some of the nice things behind the scenes that we did with solr:
- created custom request handlers that have category to facet to attribute
caching built in
- category to facet management
- ability to manage facet groups (attributes within a set facet) and assign
them to categories
- ability to create any category structure and share facet groups
- facet inheritance for any category (a facet group can be defined on a
parent category and pushed down to all children)
- ability to create sub-categories as facets instead of normal sub
categories
- simple xml configuration for the final outputted category configuration
file
I'm sure there are more cool things but that is all for now. Join the
mailing list to see more improvements in the future.
Also.. how do I get added to the Using Solr wiki page?
Thanks,
Mike Austin
RE: Requests per second/minute monitor?
Posted by Will Johnson <wj...@GETCONNECTED.COM>.
Adding entries to RequestHandlerBase.getStatistics() sounds like it
might be a reasonable compromise; backwards compatibility is kept in
place but everything from now on gets the added advantages of more
tracking. So far I've added (because I need)
avgTimePerRequest
avgRequestsPerSecond
I agree that you can pull this kind of information from logs but jmx
accessibility + the built in stats page make life a good bit simpler
especially in distributed environments.
Are there any others that people think would be nice to have or should I
stop all together?
- will
-----Original Message-----
From: Mike Klaas [mailto:mike.klaas@gmail.com]
Sent: Tuesday, May 15, 2007 5:04 PM
To: solr-user@lucene.apache.org
Subject: Re: Requests per second/minute monitor?
On 15-May-07, at 1:41 PM, Chris Hostetter wrote:
> : I'm happy to make the changes and supply a patch to move the
> logic as
> : well as adding a few simple metrics unless enough people on this
> thread
> : really feel that it's always better to do it with log files and
> : postmortem math.
>
> moving the logic would be bad .. adding new helper utilities to the
> RequestHandler Base for handlers that want to do more sounds fine
> to me.
>
> see also http://issues.apache.org/jira/browse/SOLR-176 which
> already adds
> a lot of timing info to requests.
Yeah, I held off on that as it seemed that timing/statistics might be
handleable on a larger scale. OTOH, it does give an easy way to
requesthandlers to insert detailed timing data in a logical place in
the output.
-Mike
Re: Requests per second/minute monitor?
Posted by Mike Klaas <mi...@gmail.com>.
On 15-May-07, at 1:41 PM, Chris Hostetter wrote:
> : I'm happy to make the changes and supply a patch to move the
> logic as
> : well as adding a few simple metrics unless enough people on this
> thread
> : really feel that it's always better to do it with log files and
> : postmortem math.
>
> moving the logic would be bad .. adding new helper utilities to the
> RequestHandler Base for handlers that want to do more sounds fine
> to me.
>
> see also http://issues.apache.org/jira/browse/SOLR-176 which
> already adds
> a lot of timing info to requests.
Yeah, I held off on that as it seemed that timing/statistics might be
handleable on a larger scale. OTOH, it does give an easy way to
requesthandlers to insert detailed timing data in a logical place in
the output.
-Mike
RE: Requests per second/minute monitor?
Posted by Chris Hostetter <ho...@fucit.org>.
: Why does SolrCore.setResponseHeaderValues(...) set the QTime (and other
: response header options) instead of having it as a function of
the main reason was to ensure that that data would *always* be there no
matter who wrote a request handler (or wether or not they subclassed the
RequestHandlerBase ... it's a backwards compatibility issue, we want to
ensure that data even if you have a custom request handler you've been
using since Solr 1.1 (or earlier)
: RequestHandlerBase? If things were tracked in the RequestHandlers you
: could add timing information there: avg query time, etc. I know some
you still can track those things, and return that data or log that data as
well ... although if you really wnated to know how long the *whole*
request took you would need to do it in teh ResponseWriter (or in the
core, after the response has been written)
: I'm happy to make the changes and supply a patch to move the logic as
: well as adding a few simple metrics unless enough people on this thread
: really feel that it's always better to do it with log files and
: postmortem math.
moving the logic would be bad .. adding new helper utilities to the
RequestHandler Base for handlers that want to do more sounds fine to me.
see also http://issues.apache.org/jira/browse/SOLR-176 which already adds
a lot of timing info to requests.
-Hoss
RE: Requests per second/minute monitor?
Posted by Will Johnson <wj...@GETCONNECTED.COM>.
I've needed similar logged information recently and I looked at the code
and had a few questions:
Why does SolrCore.setResponseHeaderValues(...) set the QTime (and other
response header options) instead of having it as a function of
RequestHandlerBase? If things were tracked in the RequestHandlers you
could add timing information there: avg query time, etc. I know some
people have argued that you can do that with logs but being able to pull
that info live via JMX/stats.jsp would make monitoring much cleaner in
environments with multiple machines on different networks. If things
are tracked in the handlers then people can add more statistics easily
to both query response headers and overall via custom handlers.
I'm happy to make the changes and supply a patch to move the logic as
well as adding a few simple metrics unless enough people on this thread
really feel that it's always better to do it with log files and
postmortem math.
- will
Re: Requests per second/minute monitor?
Posted by Yonik Seeley <yo...@apache.org>.
On 5/10/07, Ian Holsman <li...@holsman.net> wrote:
> What I would like to know is (and excuse the newbieness of the question) how
> to enable solr to log a file with the following data.
>
>
> - time spent (ms) in the request.
currently logged
> - IP# of the incoming request
normally in the container access log?
> - what the request was (and what handler executed it)
currently logged
> - a status code to signal if the request failed for some reasons
currently logged
> - number of rows fetched
The number of documents that matched? That's a higher level concept
rather specific to a request handler. That info is returned in most
responses though.
> and
> - the number of rows actually returned
That's also in the response, but would be largely meaningless in general.
One could also determine this number from the input parameters and the
number of docs that matched.
A better number might be size of the response (which is normally in
the container access log).
fields could be very small, or very large, and faceting, highlighting,
or other data could dwarf the size/speed due to the main response
documents.
-Yonik
ktnxbye
Posted by Ma...@ibsbe.be.
hey
we (me and my fellow student) have been on an intership the last months
we had to make a search function for an online shop and we used Solr
since the shop is not openly available for everybody, we made a demo video
for everyone to enjoy
we'ld like to thank al the people at Solr/Lucene and everyone who helped
us out over here
enjoy the show : http://deepthought.be/demovid.wmv
ktnxbye!
Re: Requests per second/minute monitor?
Posted by Ian Holsman <li...@holsman.net>.
Walter Underwood wrote:
> Yes, that is possible, but we also monitor Apache, Tomcat, the JVM, and
> OS through JMX and other live monitoring interfaces. Why invent a real-time
> HTTP log analysis system when I can fetch /search/stats.jsp at any time?
>
"there are lies, damnd lies, and statistics"
The stats page, while useful, can be dangerous if you don't fully
understand what you are looking at and it's limitations.
3 examples spring to mind.
1. a development group fetching 20k of records at a time every hour or
so off a production server.
This would represent itself on by the average rows returned going to 10.
So at the casual glance,
it looked like every request was returning 10, and the dev request was
hidden.
2. the average isn't broken down enough. certain queries are more
complex than others, and usually executed
less often than the simple ones. These skew the numbers, and hide the
bad performers
3. variation is the key to performance, not averages IMHO. I would
prefer responses delivered slower, but with no
variations (ie it will always take 2ms), then generally faster queries
which bite every once and a while. Averages
dont show you this.
ok.. thats my rant about averages gone. now of course you could always
implement variation and percentile performance
figures inside of SolR and show them on the stats page, but this can get
complex quickly.
> By "number of rows fetched", do you mean "number of documents matched"?
>
yep, I just grabbed the log format of a similar search appliance we use,
It would be better to add solr/lucene specific
things to it. I'm not tied to it.
> The log you describe is pretty useful. Ultraseek has something similar
> and that is the log most often used by admins. I'd recommend also
> logging the start and rows part of the request so you can distinguish
> between new queries and second page requests. If possible, make the
> timestamp the same as the HTTP access log so you can correlate the
> entries.
>
yep, that's a good idea as well.
I tinkered a bit more with the idea last night and came up with
https://issues.apache.org/jira/browse/SOLR-232
which allows a tomcat container access to somewhere we can put the data
we need from the request.
feedback is welcome, as I defiantly don't have all the answers, and am a
newbie when it comes to solr ;-)
--Ian
> wunder
>
> On 5/9/07 9:43 PM, "Ian Holsman" <li...@holsman.net> wrote:
>
>> Walter Underwood wrote:
>>
>>> This is for monitoring -- what happened in the last 30 seconds.
>>> Log file analysis doesn't really do that.
>>>
>> I would respectfully disagree.
>> Log file analysis of each request can give you that, and a whole lot more.
>>
>> you could either grab the stats via a regular cron job, or create a separate
>> filter to parse them real time.
>> It would then let you grab more sophisticated stats if you choose to.
>>
>> What I would like to know is (and excuse the newbieness of the question) how
>> to enable solr to log a file with the following data.
>>
>> - time spent (ms) in the request.
>> - IP# of the incoming request
>> - what the request was (and what handler executed it)
>> - a status code to signal if the request failed for some reasons
>> - number of rows fetched
>> and
>> - the number of rows actually returned
>>
>> is this possible? (I'm using tomcat if that changes the answer).
>>
>> regards
>> Ian
>>
>
>
>
Re: Requests per second/minute monitor?
Posted by Walter Underwood <wu...@netflix.com>.
Yes, that is possible, but we also monitor Apache, Tomcat, the JVM, and
OS through JMX and other live monitoring interfaces. Why invent a real-time
HTTP log analysis system when I can fetch /search/stats.jsp at any time?
By "number of rows fetched", do you mean "number of documents matched"?
The log you describe is pretty useful. Ultraseek has something similar
and that is the log most often used by admins. I'd recommend also
logging the start and rows part of the request so you can distinguish
between new queries and second page requests. If possible, make the
timestamp the same as the HTTP access log so you can correlate the
entries.
wunder
On 5/9/07 9:43 PM, "Ian Holsman" <li...@holsman.net> wrote:
>
> Walter Underwood wrote:
>> This is for monitoring -- what happened in the last 30 seconds.
>> Log file analysis doesn't really do that.
>
> I would respectfully disagree.
> Log file analysis of each request can give you that, and a whole lot more.
>
> you could either grab the stats via a regular cron job, or create a separate
> filter to parse them real time.
> It would then let you grab more sophisticated stats if you choose to.
>
> What I would like to know is (and excuse the newbieness of the question) how
> to enable solr to log a file with the following data.
>
> - time spent (ms) in the request.
> - IP# of the incoming request
> - what the request was (and what handler executed it)
> - a status code to signal if the request failed for some reasons
> - number of rows fetched
> and
> - the number of rows actually returned
>
> is this possible? (I'm using tomcat if that changes the answer).
>
> regards
> Ian