You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Manuel Le Normand <ma...@gmail.com> on 2013/09/08 17:56:09 UTC
Profiling Solr Lucene for query
Hello all
Looking on the 10% slowest queries, I get very bad performances (~60 sec
per query).
These queries have lots of conditions on my main field (more than a
hundred), including phrase queries and rows=1000. I do return only id's
though.
I can quite firmly say that this bad performance is due to slow storage
issue (that are beyond my control for now). Despite this I want to improve
my performances.
As tought in school, I started profiling these queries and the data of ~1
minute profile is located here:
http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
Main observation: most of the time I do wait for readVInt, who's stacktrace
(2 out of 2 thread dumps) is:
catalina-exec-3870 - Thread t@6615
java.lang.Thread.State: RUNNABLE
at org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
at
org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
2357)
at
ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
at org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
at
org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
at org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
at
org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
at
oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
at
org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
at
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
So I do actually wait for IO as expected, but I might be too many time page
faulting while looking for the TermBlocks (tim file), ie locating the term.
As I reindex now, would it be useful lowering down the termInterval
(default to 128)? As the FST (tip files) are that small (few 10-100 MB) so
there are no memory contentions, could I lower down this param to 8 for
example? The benefit from lowering down the term interval would be to
obligate the FST to get on memory (JVM - thanks to the NRTCachingDirectory)
as I do not control the term dictionary file (OS caching, loads an average
of 6% of it).
General configs:
solr 4.3
36 shards, each has few million docs
These 36 servers (each server has 2 replicas) are running virtual, 16GB
memory each (4GB for JVM, 12GB remain for the OS caching), consuming 260GB
of disk mounted for the index files.
Re: Profiling Solr Lucene for query
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello Manuel,
1 minute sampling brings too few data. Lowering termindex should help,
however I don't know how FST really behaves on in. It definitely helped at
3.x;
Would you mind if I ask which OS you have and which Directory
implementation is used actually?
On Sun, Sep 8, 2013 at 7:56 PM, Manuel Le Normand <
manuel.lenormand@gmail.com> wrote:
> Hello all
> Looking on the 10% slowest queries, I get very bad performances (~60 sec
> per query).
> These queries have lots of conditions on my main field (more than a
> hundred), including phrase queries and rows=1000. I do return only id's
> though.
> I can quite firmly say that this bad performance is due to slow storage
> issue (that are beyond my control for now). Despite this I want to improve
> my performances.
>
> As tought in school, I started profiling these queries and the data of ~1
> minute profile is located here:
> http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
>
> Main observation: most of the time I do wait for readVInt, who's stacktrace
> (2 out of 2 thread dumps) is:
>
> catalina-exec-3870 - Thread t@6615
> java.lang.Thread.State: RUNNABLE
> at org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
> at
>
> org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
> 2357)
> at
>
> ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
> at org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
> at
>
> org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
> at org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
> at
>
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> at
>
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> at
> oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> at
>
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> at
>
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
>
>
> So I do actually wait for IO as expected, but I might be too many time page
> faulting while looking for the TermBlocks (tim file), ie locating the term.
> As I reindex now, would it be useful lowering down the termInterval
> (default to 128)? As the FST (tip files) are that small (few 10-100 MB) so
> there are no memory contentions, could I lower down this param to 8 for
> example? The benefit from lowering down the term interval would be to
> obligate the FST to get on memory (JVM - thanks to the NRTCachingDirectory)
> as I do not control the term dictionary file (OS caching, loads an average
> of 6% of it).
>
>
> General configs:
> solr 4.3
> 36 shards, each has few million docs
> These 36 servers (each server has 2 replicas) are running virtual, 16GB
> memory each (4GB for JVM, 12GB remain for the OS caching), consuming 260GB
> of disk mounted for the index files.
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: Profiling Solr Lucene for query
Posted by Shawn Heisey <so...@elyograg.org>.
On 10/1/2013 4:04 PM, Isaac Hebsh wrote:
> Hi Shawn,
> I know that every node operates as a frontend. This is the way our cluster
> currently run.
>
> If I seperate the frontend from the nodes which hold the shards, I can let
> him different amount of CPUs as RAM. (e.g. large amount of RAM to JVM,
> because this server won't need the OS cache for reading the index, or more
> CPUs because the merging process might be more CPU intensive).
>
> Isn't it possible?
Not with SolrCloud. If you manage all your shards and replicas yourself
and use "manual" distributed search, then you can do what you're trying
to do. You lose a *LOT* of automation that SolrCloud handles for you if
you follow this route, though.
I can't find an existing feature request issue for doing this with
SolrCloud. It's a good idea, just not possible currently.
Thanks,
Shawn
Re: Profiling Solr Lucene for query
Posted by Isaac Hebsh <is...@gmail.com>.
Hi Shawn,
I know that every node operates as a frontend. This is the way our cluster
currently run.
If I seperate the frontend from the nodes which hold the shards, I can let
him different amount of CPUs as RAM. (e.g. large amount of RAM to JVM,
because this server won't need the OS cache for reading the index, or more
CPUs because the merging process might be more CPU intensive).
Isn't it possible?
On Wed, Oct 2, 2013 at 12:42 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 10/1/2013 2:35 PM, Isaac Hebsh wrote:
>
>> Hi Dmitry,
>>
>> I'm trying to examine your suggestion to create a frontend node. It sounds
>> pretty usefull.
>> I saw that every node in solr cluster can serve request for any
>> collection,
>> even if it does not hold a core of that collection. because of that, I
>> thought that adding a new node to the cluster (aka, the frontend/gateway
>> server), and creating a dummy collection (with 1 dummy core), will solve
>> the problem.
>>
>> But, I see that a request which sent to the gateway node, is not then sent
>> to the shards. Instead, the request is proxyed to a (random) core of the
>> requested collection, and from there it is sent to the shards. (It is
>> reasonable, because the SolrCore on the gateway might run with different
>> configuration, etc). This means that my new node isn't functioning as a
>> frontend (which responsible for sorting, etc.), but as a poor load
>> balancer. No performance improvement will come from this implementation.
>>
>> So, how do you suggest to implement a frontend? On the one hand, it has to
>> run a core of the target collection, but on the other hand, we don't want
>> it to hold any shard contents.
>>
>
> With SolrCloud, every node is a frontend node. If you're running
> SolrCloud, then it doesn't make sense to try and use that concept.
>
> It only makes sense to create a frontend node (or core) if you are using
> traditional distributed search, where you need to include a shards
> parameter.
>
> http://wiki.apache.org/solr/**DistributedSearch<http://wiki.apache.org/solr/DistributedSearch>
>
> Thanks,
> Shawn
>
>
Re: Profiling Solr Lucene for query
Posted by Manuel Le Normand <ma...@gmail.com>.
Would adding a dummy shard instead of a dummy collection would resolve the
situation? - e.g. editing clusterstate.json from a zookeeper client and
adding a shard with a 0-range so no docs are routed to this core. This core
would be on a separate server and act as the collection gateway.
Re: Profiling Solr Lucene for query
Posted by Dmitry Kan <so...@gmail.com>.
What Shawn has described is exactly what we do: classical distributed
no-SolrCloud setup. This is why it was possible to implement a custom
frontend solr instance.
On Wed, Oct 2, 2013 at 12:42 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 10/1/2013 2:35 PM, Isaac Hebsh wrote:
>
>> Hi Dmitry,
>>
>> I'm trying to examine your suggestion to create a frontend node. It sounds
>> pretty usefull.
>> I saw that every node in solr cluster can serve request for any
>> collection,
>> even if it does not hold a core of that collection. because of that, I
>> thought that adding a new node to the cluster (aka, the frontend/gateway
>> server), and creating a dummy collection (with 1 dummy core), will solve
>> the problem.
>>
>> But, I see that a request which sent to the gateway node, is not then sent
>> to the shards. Instead, the request is proxyed to a (random) core of the
>> requested collection, and from there it is sent to the shards. (It is
>> reasonable, because the SolrCore on the gateway might run with different
>> configuration, etc). This means that my new node isn't functioning as a
>> frontend (which responsible for sorting, etc.), but as a poor load
>> balancer. No performance improvement will come from this implementation.
>>
>> So, how do you suggest to implement a frontend? On the one hand, it has to
>> run a core of the target collection, but on the other hand, we don't want
>> it to hold any shard contents.
>>
>
> With SolrCloud, every node is a frontend node. If you're running
> SolrCloud, then it doesn't make sense to try and use that concept.
>
> It only makes sense to create a frontend node (or core) if you are using
> traditional distributed search, where you need to include a shards
> parameter.
>
> http://wiki.apache.org/solr/**DistributedSearch<http://wiki.apache.org/solr/DistributedSearch>
>
> Thanks,
> Shawn
>
>
Re: Profiling Solr Lucene for query
Posted by Shawn Heisey <so...@elyograg.org>.
On 10/1/2013 2:35 PM, Isaac Hebsh wrote:
> Hi Dmitry,
>
> I'm trying to examine your suggestion to create a frontend node. It sounds
> pretty usefull.
> I saw that every node in solr cluster can serve request for any collection,
> even if it does not hold a core of that collection. because of that, I
> thought that adding a new node to the cluster (aka, the frontend/gateway
> server), and creating a dummy collection (with 1 dummy core), will solve
> the problem.
>
> But, I see that a request which sent to the gateway node, is not then sent
> to the shards. Instead, the request is proxyed to a (random) core of the
> requested collection, and from there it is sent to the shards. (It is
> reasonable, because the SolrCore on the gateway might run with different
> configuration, etc). This means that my new node isn't functioning as a
> frontend (which responsible for sorting, etc.), but as a poor load
> balancer. No performance improvement will come from this implementation.
>
> So, how do you suggest to implement a frontend? On the one hand, it has to
> run a core of the target collection, but on the other hand, we don't want
> it to hold any shard contents.
With SolrCloud, every node is a frontend node. If you're running
SolrCloud, then it doesn't make sense to try and use that concept.
It only makes sense to create a frontend node (or core) if you are using
traditional distributed search, where you need to include a shards
parameter.
http://wiki.apache.org/solr/DistributedSearch
Thanks,
Shawn
Re: Profiling Solr Lucene for query
Posted by Isaac Hebsh <is...@gmail.com>.
Hi Dmitry,
I'm trying to examine your suggestion to create a frontend node. It sounds
pretty usefull.
I saw that every node in solr cluster can serve request for any collection,
even if it does not hold a core of that collection. because of that, I
thought that adding a new node to the cluster (aka, the frontend/gateway
server), and creating a dummy collection (with 1 dummy core), will solve
the problem.
But, I see that a request which sent to the gateway node, is not then sent
to the shards. Instead, the request is proxyed to a (random) core of the
requested collection, and from there it is sent to the shards. (It is
reasonable, because the SolrCore on the gateway might run with different
configuration, etc). This means that my new node isn't functioning as a
frontend (which responsible for sorting, etc.), but as a poor load
balancer. No performance improvement will come from this implementation.
So, how do you suggest to implement a frontend? On the one hand, it has to
run a core of the target collection, but on the other hand, we don't want
it to hold any shard contents.
On Fri, Sep 13, 2013 at 1:08 PM, Dmitry Kan <so...@gmail.com> wrote:
> Manuel,
>
> Whether to have the front end solr as aggregator of shard results depends
> on your requirements. To repeat, we found merging from many shards very
> inefficient fo our use case. It can be the opposite for you (i.e. requires
> testing). There are some limitations with distributed search, see here:
>
> http://docs.lucidworks.com/display/solr/Distributed+Search+with+Index+Sharding
>
>
> On Wed, Sep 11, 2013 at 3:35 PM, Manuel Le Normand <
> manuel.lenormand@gmail.com> wrote:
>
> > Dmitry - currently we don't have such a front end, this sounds like a
> good
> > idea creating it. And yes, we do query all 36 shards every query.
> >
> > Mikhail - I do think 1 minute is enough data, as during this exact
> minute I
> > had a single query running (that took a qtime of 1 minute). I wanted to
> > isolate these hard queries. I repeated this profiling few times.
> >
> > I think I will take the termInterval from 128 to 32 and check the
> results.
> > I'm currently using NRTCachingDirectoryFactory
> >
> >
> >
> >
> > On Mon, Sep 9, 2013 at 11:29 PM, Dmitry Kan <so...@gmail.com>
> wrote:
> >
> > > Hi Manuel,
> > >
> > > The frontend solr instance is the one that does not have its own index
> > and
> > > is doing merging of the results. Is this the case? If yes, are all 36
> > > shards always queried?
> > >
> > > Dmitry
> > >
> > >
> > > On Mon, Sep 9, 2013 at 10:11 PM, Manuel Le Normand <
> > > manuel.lenormand@gmail.com> wrote:
> > >
> > > > Hi Dmitry,
> > > >
> > > > I have solr 4.3 and every query is distributed and merged back for
> > > ranking
> > > > purpose.
> > > >
> > > > What do you mean by frontend solr?
> > > >
> > > >
> > > > On Mon, Sep 9, 2013 at 2:12 PM, Dmitry Kan <so...@gmail.com>
> > wrote:
> > > >
> > > > > are you querying your shards via a frontend solr? We have noticed,
> > that
> > > > > querying becomes much faster if results merging can be avoided.
> > > > >
> > > > > Dmitry
> > > > >
> > > > >
> > > > > On Sun, Sep 8, 2013 at 6:56 PM, Manuel Le Normand <
> > > > > manuel.lenormand@gmail.com> wrote:
> > > > >
> > > > > > Hello all
> > > > > > Looking on the 10% slowest queries, I get very bad performances
> > (~60
> > > > sec
> > > > > > per query).
> > > > > > These queries have lots of conditions on my main field (more
> than a
> > > > > > hundred), including phrase queries and rows=1000. I do return
> only
> > > id's
> > > > > > though.
> > > > > > I can quite firmly say that this bad performance is due to slow
> > > storage
> > > > > > issue (that are beyond my control for now). Despite this I want
> to
> > > > > improve
> > > > > > my performances.
> > > > > >
> > > > > > As tought in school, I started profiling these queries and the
> data
> > > of
> > > > ~1
> > > > > > minute profile is located here:
> > > > > >
> > http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
> > > > > >
> > > > > > Main observation: most of the time I do wait for readVInt, who's
> > > > > stacktrace
> > > > > > (2 out of 2 thread dumps) is:
> > > > > >
> > > > > > catalina-exec-3870 - Thread t@6615
> > > > > > java.lang.Thread.State: RUNNABLE
> > > > > > at
> org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
> > > > > > 2357)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
> > > > > > at
> org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
> > > > > > at
> > > > >
> > org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > > > at
> > > > > >
> > > >
> > org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > > > at
> > > > > >
> > > >
> > oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > > > at
> > > > > >
> > > >
> > org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
> > > > > > at
> > > > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
> > > > > >
> > > > > >
> > > > > > So I do actually wait for IO as expected, but I might be too many
> > > time
> > > > > page
> > > > > > faulting while looking for the TermBlocks (tim file), ie locating
> > the
> > > > > term.
> > > > > > As I reindex now, would it be useful lowering down the
> termInterval
> > > > > > (default to 128)? As the FST (tip files) are that small (few
> 10-100
> > > MB)
> > > > > so
> > > > > > there are no memory contentions, could I lower down this param
> to 8
> > > for
> > > > > > example? The benefit from lowering down the term interval would
> be
> > to
> > > > > > obligate the FST to get on memory (JVM - thanks to the
> > > > > NRTCachingDirectory)
> > > > > > as I do not control the term dictionary file (OS caching, loads
> an
> > > > > average
> > > > > > of 6% of it).
> > > > > >
> > > > > >
> > > > > > General configs:
> > > > > > solr 4.3
> > > > > > 36 shards, each has few million docs
> > > > > > These 36 servers (each server has 2 replicas) are running
> virtual,
> > > 16GB
> > > > > > memory each (4GB for JVM, 12GB remain for the OS caching),
> > consuming
> > > > > 260GB
> > > > > > of disk mounted for the index files.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Profiling Solr Lucene for query
Posted by Dmitry Kan <so...@gmail.com>.
Manuel,
Whether to have the front end solr as aggregator of shard results depends
on your requirements. To repeat, we found merging from many shards very
inefficient fo our use case. It can be the opposite for you (i.e. requires
testing). There are some limitations with distributed search, see here:
http://docs.lucidworks.com/display/solr/Distributed+Search+with+Index+Sharding
On Wed, Sep 11, 2013 at 3:35 PM, Manuel Le Normand <
manuel.lenormand@gmail.com> wrote:
> Dmitry - currently we don't have such a front end, this sounds like a good
> idea creating it. And yes, we do query all 36 shards every query.
>
> Mikhail - I do think 1 minute is enough data, as during this exact minute I
> had a single query running (that took a qtime of 1 minute). I wanted to
> isolate these hard queries. I repeated this profiling few times.
>
> I think I will take the termInterval from 128 to 32 and check the results.
> I'm currently using NRTCachingDirectoryFactory
>
>
>
>
> On Mon, Sep 9, 2013 at 11:29 PM, Dmitry Kan <so...@gmail.com> wrote:
>
> > Hi Manuel,
> >
> > The frontend solr instance is the one that does not have its own index
> and
> > is doing merging of the results. Is this the case? If yes, are all 36
> > shards always queried?
> >
> > Dmitry
> >
> >
> > On Mon, Sep 9, 2013 at 10:11 PM, Manuel Le Normand <
> > manuel.lenormand@gmail.com> wrote:
> >
> > > Hi Dmitry,
> > >
> > > I have solr 4.3 and every query is distributed and merged back for
> > ranking
> > > purpose.
> > >
> > > What do you mean by frontend solr?
> > >
> > >
> > > On Mon, Sep 9, 2013 at 2:12 PM, Dmitry Kan <so...@gmail.com>
> wrote:
> > >
> > > > are you querying your shards via a frontend solr? We have noticed,
> that
> > > > querying becomes much faster if results merging can be avoided.
> > > >
> > > > Dmitry
> > > >
> > > >
> > > > On Sun, Sep 8, 2013 at 6:56 PM, Manuel Le Normand <
> > > > manuel.lenormand@gmail.com> wrote:
> > > >
> > > > > Hello all
> > > > > Looking on the 10% slowest queries, I get very bad performances
> (~60
> > > sec
> > > > > per query).
> > > > > These queries have lots of conditions on my main field (more than a
> > > > > hundred), including phrase queries and rows=1000. I do return only
> > id's
> > > > > though.
> > > > > I can quite firmly say that this bad performance is due to slow
> > storage
> > > > > issue (that are beyond my control for now). Despite this I want to
> > > > improve
> > > > > my performances.
> > > > >
> > > > > As tought in school, I started profiling these queries and the data
> > of
> > > ~1
> > > > > minute profile is located here:
> > > > >
> http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
> > > > >
> > > > > Main observation: most of the time I do wait for readVInt, who's
> > > > stacktrace
> > > > > (2 out of 2 thread dumps) is:
> > > > >
> > > > > catalina-exec-3870 - Thread t@6615
> > > > > java.lang.Thread.State: RUNNABLE
> > > > > at org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
> > > > > 2357)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
> > > > > at org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
> > > > > at
> > > >
> org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > > at
> > > > >
> > >
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > > at
> > > > >
> > >
> oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > > at
> > > > >
> > >
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
> > > > > at
> > > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
> > > > >
> > > > >
> > > > > So I do actually wait for IO as expected, but I might be too many
> > time
> > > > page
> > > > > faulting while looking for the TermBlocks (tim file), ie locating
> the
> > > > term.
> > > > > As I reindex now, would it be useful lowering down the termInterval
> > > > > (default to 128)? As the FST (tip files) are that small (few 10-100
> > MB)
> > > > so
> > > > > there are no memory contentions, could I lower down this param to 8
> > for
> > > > > example? The benefit from lowering down the term interval would be
> to
> > > > > obligate the FST to get on memory (JVM - thanks to the
> > > > NRTCachingDirectory)
> > > > > as I do not control the term dictionary file (OS caching, loads an
> > > > average
> > > > > of 6% of it).
> > > > >
> > > > >
> > > > > General configs:
> > > > > solr 4.3
> > > > > 36 shards, each has few million docs
> > > > > These 36 servers (each server has 2 replicas) are running virtual,
> > 16GB
> > > > > memory each (4GB for JVM, 12GB remain for the OS caching),
> consuming
> > > > 260GB
> > > > > of disk mounted for the index files.
> > > > >
> > > >
> > >
> >
>
Re: Profiling Solr Lucene for query
Posted by Manuel Le Normand <ma...@gmail.com>.
Dmitry - currently we don't have such a front end, this sounds like a good
idea creating it. And yes, we do query all 36 shards every query.
Mikhail - I do think 1 minute is enough data, as during this exact minute I
had a single query running (that took a qtime of 1 minute). I wanted to
isolate these hard queries. I repeated this profiling few times.
I think I will take the termInterval from 128 to 32 and check the results.
I'm currently using NRTCachingDirectoryFactory
On Mon, Sep 9, 2013 at 11:29 PM, Dmitry Kan <so...@gmail.com> wrote:
> Hi Manuel,
>
> The frontend solr instance is the one that does not have its own index and
> is doing merging of the results. Is this the case? If yes, are all 36
> shards always queried?
>
> Dmitry
>
>
> On Mon, Sep 9, 2013 at 10:11 PM, Manuel Le Normand <
> manuel.lenormand@gmail.com> wrote:
>
> > Hi Dmitry,
> >
> > I have solr 4.3 and every query is distributed and merged back for
> ranking
> > purpose.
> >
> > What do you mean by frontend solr?
> >
> >
> > On Mon, Sep 9, 2013 at 2:12 PM, Dmitry Kan <so...@gmail.com> wrote:
> >
> > > are you querying your shards via a frontend solr? We have noticed, that
> > > querying becomes much faster if results merging can be avoided.
> > >
> > > Dmitry
> > >
> > >
> > > On Sun, Sep 8, 2013 at 6:56 PM, Manuel Le Normand <
> > > manuel.lenormand@gmail.com> wrote:
> > >
> > > > Hello all
> > > > Looking on the 10% slowest queries, I get very bad performances (~60
> > sec
> > > > per query).
> > > > These queries have lots of conditions on my main field (more than a
> > > > hundred), including phrase queries and rows=1000. I do return only
> id's
> > > > though.
> > > > I can quite firmly say that this bad performance is due to slow
> storage
> > > > issue (that are beyond my control for now). Despite this I want to
> > > improve
> > > > my performances.
> > > >
> > > > As tought in school, I started profiling these queries and the data
> of
> > ~1
> > > > minute profile is located here:
> > > > http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
> > > >
> > > > Main observation: most of the time I do wait for readVInt, who's
> > > stacktrace
> > > > (2 out of 2 thread dumps) is:
> > > >
> > > > catalina-exec-3870 - Thread t@6615
> > > > java.lang.Thread.State: RUNNABLE
> > > > at org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
> > > > at
> > > >
> > > >
> > >
> >
> org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
> > > > 2357)
> > > > at
> > > >
> > > >
> > >
> >
> ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
> > > > at org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
> > > > at
> > > org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > at
> > > >
> > org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > at
> > > >
> > oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > > at
> > > >
> > org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
> > > > at
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
> > > >
> > > >
> > > > So I do actually wait for IO as expected, but I might be too many
> time
> > > page
> > > > faulting while looking for the TermBlocks (tim file), ie locating the
> > > term.
> > > > As I reindex now, would it be useful lowering down the termInterval
> > > > (default to 128)? As the FST (tip files) are that small (few 10-100
> MB)
> > > so
> > > > there are no memory contentions, could I lower down this param to 8
> for
> > > > example? The benefit from lowering down the term interval would be to
> > > > obligate the FST to get on memory (JVM - thanks to the
> > > NRTCachingDirectory)
> > > > as I do not control the term dictionary file (OS caching, loads an
> > > average
> > > > of 6% of it).
> > > >
> > > >
> > > > General configs:
> > > > solr 4.3
> > > > 36 shards, each has few million docs
> > > > These 36 servers (each server has 2 replicas) are running virtual,
> 16GB
> > > > memory each (4GB for JVM, 12GB remain for the OS caching), consuming
> > > 260GB
> > > > of disk mounted for the index files.
> > > >
> > >
> >
>
Re: Profiling Solr Lucene for query
Posted by Dmitry Kan <so...@gmail.com>.
Hi Manuel,
The frontend solr instance is the one that does not have its own index and
is doing merging of the results. Is this the case? If yes, are all 36
shards always queried?
Dmitry
On Mon, Sep 9, 2013 at 10:11 PM, Manuel Le Normand <
manuel.lenormand@gmail.com> wrote:
> Hi Dmitry,
>
> I have solr 4.3 and every query is distributed and merged back for ranking
> purpose.
>
> What do you mean by frontend solr?
>
>
> On Mon, Sep 9, 2013 at 2:12 PM, Dmitry Kan <so...@gmail.com> wrote:
>
> > are you querying your shards via a frontend solr? We have noticed, that
> > querying becomes much faster if results merging can be avoided.
> >
> > Dmitry
> >
> >
> > On Sun, Sep 8, 2013 at 6:56 PM, Manuel Le Normand <
> > manuel.lenormand@gmail.com> wrote:
> >
> > > Hello all
> > > Looking on the 10% slowest queries, I get very bad performances (~60
> sec
> > > per query).
> > > These queries have lots of conditions on my main field (more than a
> > > hundred), including phrase queries and rows=1000. I do return only id's
> > > though.
> > > I can quite firmly say that this bad performance is due to slow storage
> > > issue (that are beyond my control for now). Despite this I want to
> > improve
> > > my performances.
> > >
> > > As tought in school, I started profiling these queries and the data of
> ~1
> > > minute profile is located here:
> > > http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
> > >
> > > Main observation: most of the time I do wait for readVInt, who's
> > stacktrace
> > > (2 out of 2 thread dumps) is:
> > >
> > > catalina-exec-3870 - Thread t@6615
> > > java.lang.Thread.State: RUNNABLE
> > > at org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
> > > at
> > >
> > >
> >
> org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
> > > 2357)
> > > at
> > >
> > >
> >
> ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
> > > at org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
> > > at
> > >
> > >
> >
> org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
> > > at
> > org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
> > > at
> > >
> > >
> >
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > at
> > >
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > at
> > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > at
> > >
> oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > at
> > >
> > >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > > at
> > >
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > > at
> > >
> > >
> >
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
> > > at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
> > >
> > >
> > > So I do actually wait for IO as expected, but I might be too many time
> > page
> > > faulting while looking for the TermBlocks (tim file), ie locating the
> > term.
> > > As I reindex now, would it be useful lowering down the termInterval
> > > (default to 128)? As the FST (tip files) are that small (few 10-100 MB)
> > so
> > > there are no memory contentions, could I lower down this param to 8 for
> > > example? The benefit from lowering down the term interval would be to
> > > obligate the FST to get on memory (JVM - thanks to the
> > NRTCachingDirectory)
> > > as I do not control the term dictionary file (OS caching, loads an
> > average
> > > of 6% of it).
> > >
> > >
> > > General configs:
> > > solr 4.3
> > > 36 shards, each has few million docs
> > > These 36 servers (each server has 2 replicas) are running virtual, 16GB
> > > memory each (4GB for JVM, 12GB remain for the OS caching), consuming
> > 260GB
> > > of disk mounted for the index files.
> > >
> >
>
Re: Profiling Solr Lucene for query
Posted by Manuel Le Normand <ma...@gmail.com>.
Hi Dmitry,
I have solr 4.3 and every query is distributed and merged back for ranking
purpose.
What do you mean by frontend solr?
On Mon, Sep 9, 2013 at 2:12 PM, Dmitry Kan <so...@gmail.com> wrote:
> are you querying your shards via a frontend solr? We have noticed, that
> querying becomes much faster if results merging can be avoided.
>
> Dmitry
>
>
> On Sun, Sep 8, 2013 at 6:56 PM, Manuel Le Normand <
> manuel.lenormand@gmail.com> wrote:
>
> > Hello all
> > Looking on the 10% slowest queries, I get very bad performances (~60 sec
> > per query).
> > These queries have lots of conditions on my main field (more than a
> > hundred), including phrase queries and rows=1000. I do return only id's
> > though.
> > I can quite firmly say that this bad performance is due to slow storage
> > issue (that are beyond my control for now). Despite this I want to
> improve
> > my performances.
> >
> > As tought in school, I started profiling these queries and the data of ~1
> > minute profile is located here:
> > http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
> >
> > Main observation: most of the time I do wait for readVInt, who's
> stacktrace
> > (2 out of 2 thread dumps) is:
> >
> > catalina-exec-3870 - Thread t@6615
> > java.lang.Thread.State: RUNNABLE
> > at org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
> > at
> >
> >
> org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
> > 2357)
> > at
> >
> >
> ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
> > at org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
> > at
> >
> >
> org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
> > at
> org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
> > at
> >
> >
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > at
> > org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > at
> >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > at
> > oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > at
> >
> >
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> > at
> > org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> > at
> >
> >
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
> > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
> >
> >
> > So I do actually wait for IO as expected, but I might be too many time
> page
> > faulting while looking for the TermBlocks (tim file), ie locating the
> term.
> > As I reindex now, would it be useful lowering down the termInterval
> > (default to 128)? As the FST (tip files) are that small (few 10-100 MB)
> so
> > there are no memory contentions, could I lower down this param to 8 for
> > example? The benefit from lowering down the term interval would be to
> > obligate the FST to get on memory (JVM - thanks to the
> NRTCachingDirectory)
> > as I do not control the term dictionary file (OS caching, loads an
> average
> > of 6% of it).
> >
> >
> > General configs:
> > solr 4.3
> > 36 shards, each has few million docs
> > These 36 servers (each server has 2 replicas) are running virtual, 16GB
> > memory each (4GB for JVM, 12GB remain for the OS caching), consuming
> 260GB
> > of disk mounted for the index files.
> >
>
Re: Profiling Solr Lucene for query
Posted by Dmitry Kan <so...@gmail.com>.
are you querying your shards via a frontend solr? We have noticed, that
querying becomes much faster if results merging can be avoided.
Dmitry
On Sun, Sep 8, 2013 at 6:56 PM, Manuel Le Normand <
manuel.lenormand@gmail.com> wrote:
> Hello all
> Looking on the 10% slowest queries, I get very bad performances (~60 sec
> per query).
> These queries have lots of conditions on my main field (more than a
> hundred), including phrase queries and rows=1000. I do return only id's
> though.
> I can quite firmly say that this bad performance is due to slow storage
> issue (that are beyond my control for now). Despite this I want to improve
> my performances.
>
> As tought in school, I started profiling these queries and the data of ~1
> minute profile is located here:
> http://picpaste.com/pics/IMG_20130908_132441-ZyrfXeTY.1378637843.jpg
>
> Main observation: most of the time I do wait for readVInt, who's stacktrace
> (2 out of 2 thread dumps) is:
>
> catalina-exec-3870 - Thread t@6615
> java.lang.Thread.State: RUNNABLE
> at org.apadhe.lucene.store.DataInput.readVInt(DataInput.java:108)
> at
>
> org.apaChe.lucene.codeosAockTreeIermsReade$FieldReader$SegmentTermsEnumFrame.loadBlock(BlockTreeTermsReader.java:
> 2357)
> at
>
> ora.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1745)
> at org.apadhe.lucene.index.TermContext.build(TermContext.java:95)
> at
>
> org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:221)
> at org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:326)
> at
>
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> at
>
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> at
> oro.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> at
>
> org.apache.lucene.searth.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:183)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
> at
>
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:675)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
>
>
> So I do actually wait for IO as expected, but I might be too many time page
> faulting while looking for the TermBlocks (tim file), ie locating the term.
> As I reindex now, would it be useful lowering down the termInterval
> (default to 128)? As the FST (tip files) are that small (few 10-100 MB) so
> there are no memory contentions, could I lower down this param to 8 for
> example? The benefit from lowering down the term interval would be to
> obligate the FST to get on memory (JVM - thanks to the NRTCachingDirectory)
> as I do not control the term dictionary file (OS caching, loads an average
> of 6% of it).
>
>
> General configs:
> solr 4.3
> 36 shards, each has few million docs
> These 36 servers (each server has 2 replicas) are running virtual, 16GB
> memory each (4GB for JVM, 12GB remain for the OS caching), consuming 260GB
> of disk mounted for the index files.
>