You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Aman Tandon <am...@gmail.com> on 2014/04/30 06:05:51 UTC
timeAllowed in not honoring
Hi,
I am using solr 4.2 with the index size of 40GB, while querying to my index
there are some queries which is taking the significant amount of time of
about 22 seconds *in the case of minmatch of 50%*. So i added a parameter
timeAllowed = 2000 in my query but this doesn't seems to be work. Please
help me out.
With Regards
Aman Tandon
Re: timeAllowed in not honoring
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/29/2014 11:43 PM, Aman Tandon wrote:
> My heap size is 14GB and i am not using solr cloud currently, 40GB index
> is replicated from master to two slaves.
>
> I read somewhere that it return the partial results which is computed by
> the query in that specified amount of time which is defined by this
> timeAllowed parameter, but it doesn't seems to happen.
Mikhail Khludnev has replied and explained why timeAllowed isn't
stopping the query and returning partial results.
A 14GB heap is quite large. If you aren't starting Solr with garbage
collection tuning parameters, long GC pauses *will* be happening, and
that will make some of your queries take a really long time. The wiki
page I sent has a section about garbage collection and a link showing
the GC tuning parameters that I use.
You didn't indicate how much total RAM you have. If your total RAM is
16GB, that's definitely not enough for a 14GB heap and a 40GB index.
32GB of total RAM might be enough, but it also might not be. A "perfect
world" RAM size for this setup would be at least 54GB -- the total of
heap plus index size, not counting the small number of megabytes that
the OS and its basic services take.
Thanks,
Shawn
Re: timeAllowed in not honoring
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2014-05-01 at 23:38 +0200, Shawn Heisey wrote:
> I was surprised to read that fc uses less memory.
I think that is an error in the documentation. Except for special cases,
such as asking for all facet values on a high cardinality field, I would
estimate that enum uses less memory than fc.
- Toke Eskildsen, State and University Library, Denmark
Re: timeAllowed in not honoring
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/1/2014 3:03 PM, Aman Tandon wrote:
> Please check that link
> http://wiki.apache.org/solr/SimpleFacetParameters#facet.method there is
> something mentioned in facet.method wiki
>
> *The default value is fc (except for BoolField which uses enum) since it
> tends to use less memory and is faster then the enumeration method when a
> field has many unique terms in the index.*
>
> So can you explain how enum is faster than default. Also we are currently
> using the solr 4.2 does that support this facet.method=enum, if not then
> which version should we pick.
>
> We are planning to move to SolrCloud with the version solr 4.7.1, so does
> this 14 GB of RAM will be sufficient? or should we increase it?
The fc method (which means fieldcache) puts all the data required to
build facets on that field into the fieldcache, and that data stays
there until the next commit or restart. If you are committing
frequently, that memory use might be wasted.
I was surprised to read that fc uses less memory. It may be very true
that the amount of memory required for a single call with
facet.method=enum is more than the amount of memory required in the
fieldcache for facet.method=fc, but that memory can be recovered as
garbage -- with the fc method, it can't be recovered. It sits there,
waiting for that facet to be used again, so it can speed it up. When
you commit and open a new searcher, it gets thrown away.
If you use a lot of different facets, the fieldcache can become HUGE
with the fc method. *If you don't do all those facets at the same time*
(a very important qualifier), you can switch to enum and the total
amount of resident heap memory required will be a lot less. There may
be a lot of garbage to collect, but the total heap requirement at any
given moment should be smaller. If you actually need to build a lot of
different facets at nearly the same time, enum may not actually help.
The enum method is actually a little slower than fc for a single run,
but the java heap characteristics for multiple runs can cause enum to be
faster in bulk. Try both and see what your results are.
Thanks,
Shawn
Re: timeAllowed in not honoring
Posted by Aman Tandon <am...@gmail.com>.
Apologies for late reply, Thanks Toke for a great explaination :)
I am new in solr so i am unaware of DocValues, so please can you explain.
With Regards
Aman Tandon
On Fri, May 2, 2014 at 1:52 PM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:
> On Thu, 2014-05-01 at 23:03 +0200, Aman Tandon wrote:
> > So can you explain how enum is faster than default.
>
> The fundamental difference is than enum iterates terms and counts how
> many of the documents associated to the terms are in the hits, while fc
> iterates all hits and updates a counter for the term associated to the
> document.
>
> A bit too simplified we have enum: terms->docs, fc: hits->terms. enum
> wins when there are relatively few unique terms and is much less
> affected by index updates than fc. As Shawn says, you are best off by
> testing.
>
> > We are planning to move to SolrCloud with the version solr 4.7.1, so does
> > this 14 GB of RAM will be sufficient? or should we increase it?
>
> Switching to SolrCloud does not change your fundamental memory
> requirements for searching. The merging part adds some overhead, but
> with a heap of 14GB, I would be surprised if that would require an
> increase.
>
> Consider using DocValues for facet fields with many unique values, for
> getting both speed and low memory usage at the cost of increased index
> size.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>
Re: timeAllowed in not honoring
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2014-05-01 at 23:03 +0200, Aman Tandon wrote:
> So can you explain how enum is faster than default.
The fundamental difference is than enum iterates terms and counts how
many of the documents associated to the terms are in the hits, while fc
iterates all hits and updates a counter for the term associated to the
document.
A bit too simplified we have enum: terms->docs, fc: hits->terms. enum
wins when there are relatively few unique terms and is much less
affected by index updates than fc. As Shawn says, you are best off by
testing.
> We are planning to move to SolrCloud with the version solr 4.7.1, so does
> this 14 GB of RAM will be sufficient? or should we increase it?
Switching to SolrCloud does not change your fundamental memory
requirements for searching. The merging part adds some overhead, but
with a heap of 14GB, I would be surprised if that would require an
increase.
Consider using DocValues for facet fields with many unique values, for
getting both speed and low memory usage at the cost of increased index
size.
- Toke Eskildsen, State and University Library, Denmark
Re: timeAllowed in not honoring
Posted by Aman Tandon <am...@gmail.com>.
Hi Shawn,
Please check that link
http://wiki.apache.org/solr/SimpleFacetParameters#facet.method there is
something mentioned in facet.method wiki
*The default value is fc (except for BoolField which uses enum) since it
tends to use less memory and is faster then the enumeration method when a
field has many unique terms in the index.*
So can you explain how enum is faster than default. Also we are currently
using the solr 4.2 does that support this facet.method=enum, if not then
which version should we pick.
We are planning to move to SolrCloud with the version solr 4.7.1, so does
this 14 GB of RAM will be sufficient? or should we increase it?
With Regards
Aman Tandon
On Thu, May 1, 2014 at 8:20 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/30/2014 5:53 PM, Aman Tandon wrote:
> > Shawn -> Yes we have some plans to move to SolrCloud, Our total index
> size
> > is 40GB with 11M of Docs, Available RAM 32GB, Allowed heap space for solr
> > is 14GB, the GC tuning parameters using in our server
> > is -XX:+UseConcMarkSweepGC -XX:+PrintGCApplicationStoppedTime
> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps.
>
> This means that you have about 18GB of RAM left over to cache a 40GB
> index. That's less than 50 percent. Every index is different, but this
> is in the ballpark of where performance problems begin. If you had 48GB
> of RAM, your performance (not counting possible GC problems) would
> likely be very good. 64GB would be ideal.
>
> Your only GC tuning is switching the collector to CMS. This won't be
> enough. When I had a config like this and heap of only 8GB, I was
> seeing GC pauses of 10 to 12 seconds.
>
> http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
>
> One question: Do you really need 14GB of heap? One of my servers has a
> total of 65GB of index (54 million docs) with a 7GB heap and 64GB of
> RAM. Currently I don't use facets, though. When I do, they will be
> enum. If you switch all your facets to enum, your heap requirements may
> go down. Decreasing the heap size will make more memory available for
> index caching.
>
> Thanks,
> Shawn
>
>
Re: timeAllowed in not honoring
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/30/2014 5:53 PM, Aman Tandon wrote:
> Shawn -> Yes we have some plans to move to SolrCloud, Our total index size
> is 40GB with 11M of Docs, Available RAM 32GB, Allowed heap space for solr
> is 14GB, the GC tuning parameters using in our server
> is -XX:+UseConcMarkSweepGC -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps.
This means that you have about 18GB of RAM left over to cache a 40GB
index. That's less than 50 percent. Every index is different, but this
is in the ballpark of where performance problems begin. If you had 48GB
of RAM, your performance (not counting possible GC problems) would
likely be very good. 64GB would be ideal.
Your only GC tuning is switching the collector to CMS. This won't be
enough. When I had a config like this and heap of only 8GB, I was
seeing GC pauses of 10 to 12 seconds.
http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
One question: Do you really need 14GB of heap? One of my servers has a
total of 65GB of index (54 million docs) with a 7GB heap and 64GB of
RAM. Currently I don't use facets, though. When I do, they will be
enum. If you switch all your facets to enum, your heap requirements may
go down. Decreasing the heap size will make more memory available for
index caching.
Thanks,
Shawn
Re: timeAllowed in not honoring
Posted by Aman Tandon <am...@gmail.com>.
Jeff -> Thanks Jeff this discussion on jira is really quite helpful. Thanks
for this.
Shawn -> Yes we have some plans to move to SolrCloud, Our total index size
is 40GB with 11M of Docs, Available RAM 32GB, Allowed heap space for solr
is 14GB, the GC tuning parameters using in our server
is -XX:+UseConcMarkSweepGC -XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps.
Mikhail Khludnev -> Thanks i will try to use facet.method=enum this will
definitely help us in improving some time.
With Regards
Aman Tandon
On Wed, Apr 30, 2014 at 8:30 PM, Jeff Wartes <jw...@whitepages.com> wrote:
>
> It¹s not just FacetComponent, here¹s the original feature ticket for
> timeAllowed:
> https://issues.apache.org/jira/browse/SOLR-502
>
>
> As I read it, timeAllowed only limits the time spent actually getting
> documents, not the time spent figuring out what data to get or how. I
> think that means the primary use-case is serving as a guard against
> excessive paging.
>
>
>
> On 4/30/14, 4:49 AM, "Mikhail Khludnev" <mk...@griddynamics.com>
> wrote:
>
> >On Wed, Apr 30, 2014 at 2:16 PM, Aman Tandon
> ><am...@gmail.com>wrote:
> >
> >> <lst name="query"> <double
> >> name="time">3337.0</double> </lst> <lst name="facet">
> >> <double name="time">6739.0</double> </lst>
> >>
> >
> >Most time is spent in facet counting. FacetComponent doesn't checks
> >timeAllowed right now. You can try to experiment with facet.method=enum or
> >even with https://issues.apache.org/jira/browse/SOLR-5725 or try to
> >distribute search with SolrCloud. AFAIK, you can't employ threads to speed
> >up multivalue facets.
> >
> >--
> >Sincerely yours
> >Mikhail Khludnev
> >Principal Engineer,
> >Grid Dynamics
> >
> ><http://www.griddynamics.com>
> > <mk...@griddynamics.com>
>
>
Re: timeAllowed in not honoring
Posted by Jeff Wartes <jw...@whitepages.com>.
It¹s not just FacetComponent, here¹s the original feature ticket for
timeAllowed:
https://issues.apache.org/jira/browse/SOLR-502
As I read it, timeAllowed only limits the time spent actually getting
documents, not the time spent figuring out what data to get or how. I
think that means the primary use-case is serving as a guard against
excessive paging.
On 4/30/14, 4:49 AM, "Mikhail Khludnev" <mk...@griddynamics.com> wrote:
>On Wed, Apr 30, 2014 at 2:16 PM, Aman Tandon
><am...@gmail.com>wrote:
>
>> <lst name="query"> <double
>> name="time">3337.0</double> </lst> <lst name="facet">
>> <double name="time">6739.0</double> </lst>
>>
>
>Most time is spent in facet counting. FacetComponent doesn't checks
>timeAllowed right now. You can try to experiment with facet.method=enum or
>even with https://issues.apache.org/jira/browse/SOLR-5725 or try to
>distribute search with SolrCloud. AFAIK, you can't employ threads to speed
>up multivalue facets.
>
>--
>Sincerely yours
>Mikhail Khludnev
>Principal Engineer,
>Grid Dynamics
>
><http://www.griddynamics.com>
> <mk...@griddynamics.com>
Re: timeAllowed in not honoring
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Wed, Apr 30, 2014 at 2:16 PM, Aman Tandon <am...@gmail.com>wrote:
> <lst name="query"> <double
> name="time">3337.0</double> </lst> <lst name="facet">
> <double name="time">6739.0</double> </lst>
>
Most time is spent in facet counting. FacetComponent doesn't checks
timeAllowed right now. You can try to experiment with facet.method=enum or
even with https://issues.apache.org/jira/browse/SOLR-5725 or try to
distribute search with SolrCloud. AFAIK, you can't employ threads to speed
up multivalue facets.
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: timeAllowed in not honoring
Posted by Aman Tandon <am...@gmail.com>.
Hi Salman,
here is the my debug query dump please help!. I am unable to find the
wildcards in it.
<?xml version="1.0" encoding="UTF-8"?><response><lst
name="responseHeader"> <bool name="partialResults">true</bool> <int
name="status">0</int> <int name="QTime">10080</int></lst><result
name="response" numFound="976303" start="0"></result><lst
name="facet_counts"> <lst name="facet_queries"/> <lst
name="facet_fields"> <lst name="city"> <int name="delhi
ncr">884159</int> <int name="delhi">629472</int> <int
name="mumbai">491426</int> <int name="ahmedabad">259356</int>
<int name="chennai">259029</int> <int
name="bengaluru">257193</int> <int name="kolkata">195077</int>
<int name="pune">193569</int> <int
name="hyderabad">179369</int> <int name="jaipur">115356</int>
<int name="coimbatore">111644</int> <int
name="noida">86794</int> <int name="surat">80621</int> <int
name="gurgaon">72815</int> <int name="rajkot">68982</int>
<int name="vadodara">65082</int> <int name="ludhiana">63244</int>
<int name="thane">55091</int> <int name="indore">50225</int>
<int name="ghaziabad">49756</int> <int
name="faridabad">45322</int> <int name="navi mumbai">40127</int>
<int name="tiruppur">37639</int> <int
name="nagpur">37126</int> <int name="kochi">32874</int> </lst>
<lst name="datatype"> <int name="product">966816</int>
<int name="offer">6003</int> <int name="company">3484</int>
</lst> </lst> <lst name="facet_dates"/> <lst
name="facet_ranges"/></lst><lst name="debug"> <str
name="rawquerystring">misc items</str> <str name="querystring">misc
items</str> <str
name="parsedquery">BoostedQuery(boost(+(((titlex:misc^1.5 |
smalldesc:misc | titlews:misc^0.5 | city:misc | usrpcatname:misc |
mcatnametext:misc^0.2)~0.3 (titlex:item^1.5 | smalldesc:item |
titlews:items^0.5 | city:items | usrpcatname:item |
mcatnametext:item^0.2)~0.3)~1) (mcatnametext:"misc item"^0.5)~0.3
(titlews:"misc items")~0.3 (titlex:"misc item"^3.0)~0.3
(smalldesc:"misc item"^2.0)~0.3 (usrpcatname:"misc item")~0.3
(),product(map(query(+(titlex:"item imsw")~0.3
(),def=0.0),0.0,0.0,1.0),map(query(+(titlex:"misc item imsw")~0.3
(),def=0.0),0.0,0.0,1.0),map(int(sdesclen),0.0,150.0,1.0),map(int(sdesclen),0.0,0.0,0.1),map(int(CustTypeWt),699.0,699.0,1.2),map(int(CustTypeWt),199.0,199.0,1.3),map(int(CustTypeWt),0.0,179.0,1.35),1.0/(3.16E-11*float(ms(const(1398852652419),date(lastactiondatet)))+1.0),map(ms(const(1398852652419),date(blpurchasedate)),0.0,2.6E9,1.15),map(query(+(attribs:hot)~0.3
(titlex:hot^3.0 | smalldesc:hot^2.0 | titlews:hot | city:hot |
usrpcatname:hot |
mcatnametext:hot^0.5)~0.3,def=0.0),0.0,0.0,1.0),map(query(+(attribs:dupimg)~0.3
(titlex:dupimg^3.0 | smalldesc:dupimg^2.0 | titlews:dupimg |
city:dupimg | usrpcatname:dupimg |
mcatnametext:dupimg^0.5)~0.3,def=0.0),0.0,0.0,1.0),map(query(+(isphoto:T)~0.3
(),def=0.0),0.0,0.0,0.1))))</str> <str
name="parsedquery_toString">boost(+(((titlex:misc^1.5 | smalldesc:misc
| titlews:misc^0.5 | city:misc | usrpcatname:misc |
mcatnametext:misc^0.2)~0.3 (titlex:item^1.5 | smalldesc:item |
titlews:items^0.5 | city:items | usrpcatname:item |
mcatnametext:item^0.2)~0.3)~1) (mcatnametext:"misc item"^0.5)~0.3
(titlews:"misc items")~0.3 (titlex:"misc item"^3.0)~0.3
(smalldesc:"misc item"^2.0)~0.3 (usrpcatname:"misc item")~0.3
(),product(map(query(+(titlex:"item imsw")~0.3
(),def=0.0),0.0,0.0,1.0),map(query(+(titlex:"misc item imsw")~0.3
(),def=0.0),0.0,0.0,1.0),map(int(sdesclen),0.0,150.0,1.0),map(int(sdesclen),0.0,0.0,0.1),map(int(CustTypeWt),699.0,699.0,1.2),map(int(CustTypeWt),199.0,199.0,1.3),map(int(CustTypeWt),0.0,179.0,1.35),1.0/(3.16E-11*float(ms(const(1398852652419),date(lastactiondatet)))+1.0),map(ms(const(1398852652419),date(blpurchasedate)),0.0,2.6E9,1.15),map(query(+(attribs:hot)~0.3
(titlex:hot^3.0 | smalldesc:hot^2.0 | titlews:hot | city:hot |
usrpcatname:hot |
mcatnametext:hot^0.5)~0.3,def=0.0),0.0,0.0,1.0),map(query(+(attribs:dupimg)~0.3
(titlex:dupimg^3.0 | smalldesc:dupimg^2.0 | titlews:dupimg |
city:dupimg | usrpcatname:dupimg |
mcatnametext:dupimg^0.5)~0.3,def=0.0),0.0,0.0,1.0),map(query(+(isphoto:T)~0.3
(),def=0.0),0.0,0.0,0.1)))</str> <lst name="explain"/> <str
name="QParser">SynonymExpandingExtendedDismaxQParser</str> <null
name="altquerystring"/> <null name="boost_queries"/> <arr
name="parsed_boost_queries"/> <null name="boostfuncs"/> <arr
name="filter_queries">
<str>{!tag=cityf}latlong:"Intersects(Circle(28.63576,77.22445
d=2.248))"</str> <str>attribs:(locprefglobal locprefnational
locprefcity)</str> <str>+((+datatype:product +attribs:(aprstatus20
aprstatus40 aprstatus50) +aggregate:true -attribs:liststatusnfl
+((+countryiso:IN +isfcp:true) CustTypeWt:[149 TO 1499]))
(+datatype:offer +iildisplayflag:true) (+datatype:company
-attribs:liststatusnfl +((+countryiso:IN +isfcp:true) CustTypeWt:[149
TO 1499]))) -attribs:liststatusdnf</str> </arr> <arr
name="parsed_filter_queries">
<str>ConstantScore(org.apache.lucene.spatial.prefix.IntersectsPrefixTreeFilter@414cd6c2)</str>
<str>attribs:locprefglobal attribs:locprefnational
attribs:locprefcity</str> <str>+((+datatype:product
+(attribs:aprstatus20 attribs:aprstatus40 attribs:aprstatus50)
+aggregate:true -attribs:liststatusnfl +((+countryiso:IN +isfcp:true)
CustTypeWt:[149 TO 1499])) (+datatype:offer +iildisplayflag:true)
(+datatype:company -attribs:liststatusnfl +((+countryiso:IN
+isfcp:true) CustTypeWt:[149 TO 1499]))) -attribs:liststatusdnf</str>
</arr> <lst name="timing"> <double name="time">10079.0</double>
<lst name="prepare"> <double name="time">3.0</double> <lst
name="query"> <double name="time">3.0</double> </lst>
<lst name="facet"> <double name="time">0.0</double> </lst>
<lst name="mlt"> <double name="time">0.0</double>
</lst> <lst name="highlight"> <double
name="time">0.0</double> </lst> <lst name="stats">
<double name="time">0.0</double> </lst> <lst
name="spellcheck"> <double name="time">0.0</double> </lst>
<lst name="debug"> <double name="time">0.0</double>
</lst> </lst> <lst name="process"> <double
name="time">10076.0</double> <lst name="query"> <double
name="time">3337.0</double> </lst> <lst name="facet">
<double name="time">6739.0</double> </lst> <lst name="mlt">
<double name="time">0.0</double> </lst> <lst
name="highlight"> <double name="time">0.0</double> </lst>
<lst name="stats"> <double name="time">0.0</double>
</lst> <lst name="spellcheck"> <double
name="time">0.0</double> </lst> <lst name="debug">
<double name="time">0.0</double> </lst> </lst>
</lst></lst></response>
With Regards
Aman Tandon
On Wed, Apr 30, 2014 at 12:09 PM, Salman Akram <
salman.akram@northbaysolutions.net> wrote:
> I had this issue too. timeAllowed only works for a certain phase of the
> query. I think that's the 'process' part. However, if the query is taking
> time in 'prepare' phase (e.g. I think for wildcards to get all the possible
> combinations before running the query) it won't have any impact on that.
> You can debug your query and confirm that.
>
>
> On Wed, Apr 30, 2014 at 10:43 AM, Aman Tandon <amantandon.10@gmail.com
> >wrote:
>
> > Shawn this is the first time i raised this problem.
> >
> > My heap size is 14GB and i am not using solr cloud currently, 40GB index
> > is replicated from master to two slaves.
> >
> > I read somewhere that it return the partial results which is computed by
> > the query in that specified amount of time which is defined by this
> > timeAllowed parameter, but it doesn't seems to happen.
> >
> > Here is the link :
> > http://wiki.apache.org/solr/CommonQueryParameters#timeAllowed
> >
> > *The time allowed for a search to finish. This value only applies to the
> > search and not to requests in general. Time is in milliseconds. Values
> <= 0
> > mean no time restriction. Partial results may be returned (if there are
> > any). *
> >
> >
> >
> > With Regards
> > Aman Tandon
> >
> >
> > On Wed, Apr 30, 2014 at 10:05 AM, Shawn Heisey <so...@elyograg.org>
> wrote:
> >
> > > On 4/29/2014 10:05 PM, Aman Tandon wrote:
> > > > I am using solr 4.2 with the index size of 40GB, while querying to my
> > > index
> > > > there are some queries which is taking the significant amount of time
> > of
> > > > about 22 seconds *in the case of minmatch of 50%*. So i added a
> > parameter
> > > > timeAllowed = 2000 in my query but this doesn't seems to be work.
> > Please
> > > > help me out.
> > >
> > > I remember reading that timeAllowed has some limitations about which
> > > stages of a query it can limit, particularly in the distributed case.
> > > These limitations mean that it cannot always limit the total time for a
> > > query. I do not remember precisely what those limitations are, and I
> > > cannot find whatever it was that I was reading.
> > >
> > > When I looked through my local list archive to see if you had ever
> > > mentioned how much RAM you have and what the size of your Solr heap is,
> > > there didn't seem to be anything. There's not enough information for
> me
> > > to know whether that 40GB is the amount of index data on a single
> > > SolrCloud server, or whether it's the total size of the index across
> all
> > > servers.
> > >
> > > If we leave timeAllowed alone for a moment and treat this purely as a
> > > performance problem, usually my questions revolve around figuring out
> > > whether you have enough RAM. Here's where that conversation ends up:
> > >
> > > http://wiki.apache.org/solr/SolrPerformanceProblems
> > >
> > > I think I've probably mentioned this to you before on another thread.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
>
>
> --
> Regards,
>
> Salman Akram
>
Re: timeAllowed in not honoring
Posted by Salman Akram <sa...@northbaysolutions.net>.
I had this issue too. timeAllowed only works for a certain phase of the
query. I think that's the 'process' part. However, if the query is taking
time in 'prepare' phase (e.g. I think for wildcards to get all the possible
combinations before running the query) it won't have any impact on that.
You can debug your query and confirm that.
On Wed, Apr 30, 2014 at 10:43 AM, Aman Tandon <am...@gmail.com>wrote:
> Shawn this is the first time i raised this problem.
>
> My heap size is 14GB and i am not using solr cloud currently, 40GB index
> is replicated from master to two slaves.
>
> I read somewhere that it return the partial results which is computed by
> the query in that specified amount of time which is defined by this
> timeAllowed parameter, but it doesn't seems to happen.
>
> Here is the link :
> http://wiki.apache.org/solr/CommonQueryParameters#timeAllowed
>
> *The time allowed for a search to finish. This value only applies to the
> search and not to requests in general. Time is in milliseconds. Values <= 0
> mean no time restriction. Partial results may be returned (if there are
> any). *
>
>
>
> With Regards
> Aman Tandon
>
>
> On Wed, Apr 30, 2014 at 10:05 AM, Shawn Heisey <so...@elyograg.org> wrote:
>
> > On 4/29/2014 10:05 PM, Aman Tandon wrote:
> > > I am using solr 4.2 with the index size of 40GB, while querying to my
> > index
> > > there are some queries which is taking the significant amount of time
> of
> > > about 22 seconds *in the case of minmatch of 50%*. So i added a
> parameter
> > > timeAllowed = 2000 in my query but this doesn't seems to be work.
> Please
> > > help me out.
> >
> > I remember reading that timeAllowed has some limitations about which
> > stages of a query it can limit, particularly in the distributed case.
> > These limitations mean that it cannot always limit the total time for a
> > query. I do not remember precisely what those limitations are, and I
> > cannot find whatever it was that I was reading.
> >
> > When I looked through my local list archive to see if you had ever
> > mentioned how much RAM you have and what the size of your Solr heap is,
> > there didn't seem to be anything. There's not enough information for me
> > to know whether that 40GB is the amount of index data on a single
> > SolrCloud server, or whether it's the total size of the index across all
> > servers.
> >
> > If we leave timeAllowed alone for a moment and treat this purely as a
> > performance problem, usually my questions revolve around figuring out
> > whether you have enough RAM. Here's where that conversation ends up:
> >
> > http://wiki.apache.org/solr/SolrPerformanceProblems
> >
> > I think I've probably mentioned this to you before on another thread.
> >
> > Thanks,
> > Shawn
> >
> >
>
--
Regards,
Salman Akram
Re: timeAllowed in not honoring
Posted by Aman Tandon <am...@gmail.com>.
Shawn this is the first time i raised this problem.
My heap size is 14GB and i am not using solr cloud currently, 40GB index
is replicated from master to two slaves.
I read somewhere that it return the partial results which is computed by
the query in that specified amount of time which is defined by this
timeAllowed parameter, but it doesn't seems to happen.
Here is the link :
http://wiki.apache.org/solr/CommonQueryParameters#timeAllowed
*The time allowed for a search to finish. This value only applies to the
search and not to requests in general. Time is in milliseconds. Values <= 0
mean no time restriction. Partial results may be returned (if there are
any). *
With Regards
Aman Tandon
On Wed, Apr 30, 2014 at 10:05 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/29/2014 10:05 PM, Aman Tandon wrote:
> > I am using solr 4.2 with the index size of 40GB, while querying to my
> index
> > there are some queries which is taking the significant amount of time of
> > about 22 seconds *in the case of minmatch of 50%*. So i added a parameter
> > timeAllowed = 2000 in my query but this doesn't seems to be work. Please
> > help me out.
>
> I remember reading that timeAllowed has some limitations about which
> stages of a query it can limit, particularly in the distributed case.
> These limitations mean that it cannot always limit the total time for a
> query. I do not remember precisely what those limitations are, and I
> cannot find whatever it was that I was reading.
>
> When I looked through my local list archive to see if you had ever
> mentioned how much RAM you have and what the size of your Solr heap is,
> there didn't seem to be anything. There's not enough information for me
> to know whether that 40GB is the amount of index data on a single
> SolrCloud server, or whether it's the total size of the index across all
> servers.
>
> If we leave timeAllowed alone for a moment and treat this purely as a
> performance problem, usually my questions revolve around figuring out
> whether you have enough RAM. Here's where that conversation ends up:
>
> http://wiki.apache.org/solr/SolrPerformanceProblems
>
> I think I've probably mentioned this to you before on another thread.
>
> Thanks,
> Shawn
>
>
Re: timeAllowed in not honoring
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/29/2014 10:05 PM, Aman Tandon wrote:
> I am using solr 4.2 with the index size of 40GB, while querying to my index
> there are some queries which is taking the significant amount of time of
> about 22 seconds *in the case of minmatch of 50%*. So i added a parameter
> timeAllowed = 2000 in my query but this doesn't seems to be work. Please
> help me out.
I remember reading that timeAllowed has some limitations about which
stages of a query it can limit, particularly in the distributed case.
These limitations mean that it cannot always limit the total time for a
query. I do not remember precisely what those limitations are, and I
cannot find whatever it was that I was reading.
When I looked through my local list archive to see if you had ever
mentioned how much RAM you have and what the size of your Solr heap is,
there didn't seem to be anything. There's not enough information for me
to know whether that 40GB is the amount of index data on a single
SolrCloud server, or whether it's the total size of the index across all
servers.
If we leave timeAllowed alone for a moment and treat this purely as a
performance problem, usually my questions revolve around figuring out
whether you have enough RAM. Here's where that conversation ends up:
http://wiki.apache.org/solr/SolrPerformanceProblems
I think I've probably mentioned this to you before on another thread.
Thanks,
Shawn