You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Samarendra Pratap <sa...@gmail.com> on 2010/04/27 14:51:37 UTC
Right memory for search application
Hi.
I am searching for some guidance on right memory options for my Search
Server application. How much memory a lucene based application should be
given?
Till a few days back I was running my search server on java 1.4 with memory
options "-Xmx3600m" which was running quite fine. After upgrading the JVM to
*java 6* we noticed a few times that our application fell idle (perhaps
hung). It was not serving the requests although the port was open. I changed
the "-Xmx" from 3600m to 5000m.
Currently it is running OK but this created a curiosity in my mind that how
to find the right memory size for a lucene based application (or any java
application). I believe it heavily depends on index size but how do I
calculate it (at least approximately) without hit & trial?
The details about my application and server are following
*[ system configuration ]*
# uname -a
Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007 x86_64
x86_64 x86_64 GNU/Linux
*[ java version ]*
# /usr/lib/jre1.6.0_20/bin/java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
*[ application command line ]*
# java -Xmx5000m -server -classpath
lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer conf/search.conf
*[ memory information ]*
*
# cat /proc/meminfo
MemTotal: 8156660 kB
MemFree: 46924 kB
Buffers: 53768 kB
Cached: 3194224 kB
SwapCached: 60 kB
Active: 6086272 kB
Inactive: 919276 kB
SwapTotal: 2096472 kB
SwapFree: 2095432 kB
Dirty: 1148 kB
Writeback: 0 kB
AnonPages: 3756496 kB
Mapped: 24800 kB
Slab: 1058124 kB
SReclaimable: 15144 kB
SUnreclaim: 1042980 kB
PageTables: 12232 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 6174800 kB
Committed_AS: 4588900 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 264904 kB
VmallocChunk: 34359473387 kB
More info*
Search Server application is running on a few servers each of which contains
same of copy of index.
Total size of index: 23 GB
Total Documents in index: 18,000,000
Maximum fields per index: 56 (all analyzed, 4 small fields stored, field
"contents" is the largest one, created from a text of as large as 5 - 8 KB)
Query frequency: 1 query per second per server (in peak hours)
Queries are taking around 800 - 1000 ms per query
*Other memory related things in Search application*
There is really nothing else which will consume noticeable memory. No
database connection is made. The application simply returns the IDs from the
index based on the query.
Any help on choosing right memory option is much appreciated.
--
Regards,
Samar
Re: Right memory for search application
Posted by Samarendra Pratap <sa...@gmail.com>.
Great explanation Erick.
Thanks. I'll try that.
On Wed, Apr 28, 2010 at 8:27 PM, Erick Erickson <er...@gmail.com>wrote:
> Quick reply to question (4). Not quite right, you're not gaining anything,
> you still have a yymmddHHMMSS field that will consume all the memory
> when you sort by it.
>
> Remember that the controlling part here is the number of unique values. So
> think about two fields, yymmdd and HHMMSS. Use the HHMMSS as the
> secondary sort and yymmdd as the primary. This will sort correctly since
> any time HHMMSS is used, the yymmdd is guaranteed be equal.
>
> And you can extend this ad nauseum. For instance, you could use 6
> fields, yy, mm, dd, HH, MM, SS and have a very small number of
> unique values in each using really tiny amounts of memory to sort down
> to the second in this case.
>
> Best
> Erick
>
> On Wed, Apr 28, 2010 at 2:24 AM, Samarendra Pratap <samarzone@gmail.com
> >wrote:
>
> > I have got a lot of valuable information in this thread so far.
> > Thanks to all.
> >
> > In my last mail I mentioned only two fields because others' usage was
> > negligible and I thought they are not important. But now after *Toke
> > *explained
> > the formulae, I think sorting on those fields would also be consuming a
> > huge
> > part of memory.There are 2 other sorting fields; one of which is used in
> > both ascending/descending sorting.
> >
> > Within next couple of days (or may be a week) I'll be
> >
> > 1. profiling my application,
> > 2. analyzing and tuning GC options
> >
> >
> >
> > However, I have a few more curiosities -
> >
> > 1. Tom wrote:
> >
> > *Have you checked that your machine is correctly identified as a server*
> > *and has optimized GC settings?*
> >
> > *I did not understand the meaning of "correctly identified as a server"
> Can
> > you please help me understand?*
> > *
> > *
> > 2. *Should I change the type of fields?*
> > ** As I said in my first mail that I have 56 fields in my index, most of
> > them contain a numeric value or one of system defined values (e.g. gender
> > field can contain only "male", "female", or "unknown"). There are only 7
> > fields which are indexed with user defined values.
> > All the fields are created with *Field*
> > (String name, String value, Field.Store store, Field.Index index)
> > It would be creating all the fields as normal string fields. Is it
> > *always*a good idea to use specific classes (NumericField, DateTime
> > etc.). We do not
> > have space problem if that matters.
> >
> > 3. *Is there any advice on number of fields?*
> > *Somewhere on the net I read that instead of keeping different type of
> > values in different fields, (e.g. field1:value1, field2:value2,...) one
> > should practice keeping different values in single field (e.g.
> > field:field1_value1,
> > field:field2_value2,...). But I could not confirm it from anywhere else.
> > Any
> > comments?*
> >
> > 4. Ian wrote:
> >
> > *Sorting by score down to the second will use a lot of memory. Can you*
> > *make it less granular?*
> >
> > Is it less painful sorting on two fields; first on yymmdd and then on
> > yymmddHHMMSS than sorting just on latter? (Naturally it should use second
> > field, only where required but technically ...?)
> >
> >
> > Thanks again for the invaluable support I am getting from here.
> >
> > - Samar
> >
> > On Wed, Apr 28, 2010 at 9:12 AM, Lance Norskog <go...@gmail.com>
> wrote:
> >
> > > Solr's timestamp representation (TrieDateField) is tuned for space and
> > > speed. It has a compressed representation, and sorts with far less
> > > space than Strings.
> > >
> > > Also you get something called a date facet, which lets you bucketize
> > > facet searches by time block.
> > >
> > > On Tue, Apr 27, 2010 at 1:02 PM, Toke Eskildsen <
> te@statsbiblioteket.dk>
> > > wrote:
> > > > Samarendra Pratap [samarzone@gmail.com] wrote:
> > > >> 1. Our default option is sort by score, however almost 8% of
> searches
> > > use
> > > >> sorting on a field (yyyymmddHHMMSS). This field is indexed as string
> > > (not as
> > > >> NumericField or DateField).
> > > >
> > > > Guessing that the timestamp is practically unique for each document,
> > > sorting by String takes up a bit more than
> > > > 18M * (40 bytes + 2 * "yyyymmddHHMMSS".length() bytes) ~= 1.2 GB of
> RAM
> > > as the Strings are cached. Coupled with the normal overhead of just
> > opening
> > > an index of your size (500MB by your measurements?), I would have
> guessed
> > > that 3600MB would definitely be enough to open the index and do sorted
> > > searches.
> > > >
> > > > I realize that fiddling with production servers is dangerous, but
> > > connecting with JConsole and forcing a garbage collection might be
> > > acceptable? That should enable you to determine whether you're leaking
> > > memory or if it's just the JVM being greedy. I'd guess you leaking
> > though,
> > > as HotSpot does not normally allocate up to the limit if it does not
> need
> > > to.
> > > >
> > > > Anyway, changing to one of the optimized fields for sorting dates
> > should
> > > shave 1 GB off the memory requirement, so I'll recommend doing that no
> > > matter what the main cause of your memory problems is.
> > > >
> > > > Regards,
> > > > Toke Eskildsen
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Lance Norskog
> > > goksron@gmail.com
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> >
> > --
> > Regards,
> > Samar
> >
>
--
Regards,
Samar
Re: Right memory for search application
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Wed, 2010-04-28 at 16:57 +0200, Erick Erickson wrote:
> And you can extend this ad nauseum. For instance, you could use 6
> fields, yy, mm, dd, HH, MM, SS and have a very small number of
> unique values in each using really tiny amounts of memory to sort down
> to the second in this case.
A reference to a String takes up 4 or 8 bytes, depending on JVM and
setup. When doing String-based sorting in Lucene, there will be a
reference for each document for each sort-field. With 18M documents,
this will be at least 18M * 4 bytes * 6 = 432 MB. Plus the negligible
amount of String objects. It is less than the previously estimated
1.2GB, but still worth noting.
- Toke
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Right memory for search application
Posted by Erick Erickson <er...@gmail.com>.
Quick reply to question (4). Not quite right, you're not gaining anything,
you still have a yymmddHHMMSS field that will consume all the memory
when you sort by it.
Remember that the controlling part here is the number of unique values. So
think about two fields, yymmdd and HHMMSS. Use the HHMMSS as the
secondary sort and yymmdd as the primary. This will sort correctly since
any time HHMMSS is used, the yymmdd is guaranteed be equal.
And you can extend this ad nauseum. For instance, you could use 6
fields, yy, mm, dd, HH, MM, SS and have a very small number of
unique values in each using really tiny amounts of memory to sort down
to the second in this case.
Best
Erick
On Wed, Apr 28, 2010 at 2:24 AM, Samarendra Pratap <sa...@gmail.com>wrote:
> I have got a lot of valuable information in this thread so far.
> Thanks to all.
>
> In my last mail I mentioned only two fields because others' usage was
> negligible and I thought they are not important. But now after *Toke
> *explained
> the formulae, I think sorting on those fields would also be consuming a
> huge
> part of memory.There are 2 other sorting fields; one of which is used in
> both ascending/descending sorting.
>
> Within next couple of days (or may be a week) I'll be
>
> 1. profiling my application,
> 2. analyzing and tuning GC options
>
>
>
> However, I have a few more curiosities -
>
> 1. Tom wrote:
>
> *Have you checked that your machine is correctly identified as a server*
> *and has optimized GC settings?*
>
> *I did not understand the meaning of "correctly identified as a server" Can
> you please help me understand?*
> *
> *
> 2. *Should I change the type of fields?*
> ** As I said in my first mail that I have 56 fields in my index, most of
> them contain a numeric value or one of system defined values (e.g. gender
> field can contain only "male", "female", or "unknown"). There are only 7
> fields which are indexed with user defined values.
> All the fields are created with *Field*
> (String name, String value, Field.Store store, Field.Index index)
> It would be creating all the fields as normal string fields. Is it
> *always*a good idea to use specific classes (NumericField, DateTime
> etc.). We do not
> have space problem if that matters.
>
> 3. *Is there any advice on number of fields?*
> *Somewhere on the net I read that instead of keeping different type of
> values in different fields, (e.g. field1:value1, field2:value2,...) one
> should practice keeping different values in single field (e.g.
> field:field1_value1,
> field:field2_value2,...). But I could not confirm it from anywhere else.
> Any
> comments?*
>
> 4. Ian wrote:
>
> *Sorting by score down to the second will use a lot of memory. Can you*
> *make it less granular?*
>
> Is it less painful sorting on two fields; first on yymmdd and then on
> yymmddHHMMSS than sorting just on latter? (Naturally it should use second
> field, only where required but technically ...?)
>
>
> Thanks again for the invaluable support I am getting from here.
>
> - Samar
>
> On Wed, Apr 28, 2010 at 9:12 AM, Lance Norskog <go...@gmail.com> wrote:
>
> > Solr's timestamp representation (TrieDateField) is tuned for space and
> > speed. It has a compressed representation, and sorts with far less
> > space than Strings.
> >
> > Also you get something called a date facet, which lets you bucketize
> > facet searches by time block.
> >
> > On Tue, Apr 27, 2010 at 1:02 PM, Toke Eskildsen <te...@statsbiblioteket.dk>
> > wrote:
> > > Samarendra Pratap [samarzone@gmail.com] wrote:
> > >> 1. Our default option is sort by score, however almost 8% of searches
> > use
> > >> sorting on a field (yyyymmddHHMMSS). This field is indexed as string
> > (not as
> > >> NumericField or DateField).
> > >
> > > Guessing that the timestamp is practically unique for each document,
> > sorting by String takes up a bit more than
> > > 18M * (40 bytes + 2 * "yyyymmddHHMMSS".length() bytes) ~= 1.2 GB of RAM
> > as the Strings are cached. Coupled with the normal overhead of just
> opening
> > an index of your size (500MB by your measurements?), I would have guessed
> > that 3600MB would definitely be enough to open the index and do sorted
> > searches.
> > >
> > > I realize that fiddling with production servers is dangerous, but
> > connecting with JConsole and forcing a garbage collection might be
> > acceptable? That should enable you to determine whether you're leaking
> > memory or if it's just the JVM being greedy. I'd guess you leaking
> though,
> > as HotSpot does not normally allocate up to the limit if it does not need
> > to.
> > >
> > > Anyway, changing to one of the optimized fields for sorting dates
> should
> > shave 1 GB off the memory requirement, so I'll recommend doing that no
> > matter what the main cause of your memory problems is.
> > >
> > > Regards,
> > > Toke Eskildsen
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> >
> >
> > --
> > Lance Norskog
> > goksron@gmail.com
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> Regards,
> Samar
>
Re: how to design Lucene Document and Field to indexing and searching email message and attachments
Posted by 刘庆志 <li...@gmail.com>.
Erick:
Thanks for your information.
I search for the similar question and get a very like issue:Best Practice: emails and file-attachments on 15 August 2006,in mailing list archives(http://mail-archives.apache.org/mod_mbox/lucene-java-user/200608.mbox/browser),but there is no final answer in that thread.
dazhi
----- Original Message -----
From: "Erick Erickson" <er...@gmail.com>
To: <ja...@lucene.apache.org>
Sent: Wednesday, April 28, 2010 11:00 PM
Subject: Re: how to design Lucene Document and Field to indexing and searching email message and attachments
This problem has been discussed several times, although I can't
remember the answer. So I'd recommend searching the mail archive
first.
Lucid maintains a searchable archive, see:
http://www.lucidimagination.com/About-Search
HTH
Erick
On Wed, Apr 28, 2010 at 2:35 AM, 刘庆志 <li...@gmail.com> wrote:
> hi all:
> our bussiness system generate some data,that information structrue like
> email message,one message have some attachments,so we can use email message
> to think of our data,I need index and search the message and its
> attachments,and when display hits,must display two kinds of links for every
> hit: one kind for message,and the other for attachments which match the
> query criteria,so the former kind there is only one link,but latter may be
> zero to n links.
> one design may be as:for every message design one Lucene Document,it has a
> field to record its id,let's name the field id,an other field to correspond
> all its attachments,let's name the field attachments,afert that we also
> design a Lucene Document to correspond every message's every attachment,this
> Document has a field record its message's id,let's name the field messageid,
> so when query,we can retrieval messages may be itself's cotent or its
> attachments content match the query criteria,for generating links for
> attachments which match the query criteria,we can requey again,this time we
> can query only the message's attachments by adding a query condition that
> messageid=father query's messge id.it's obviously,there are two
> disadvantages: 1,it indexes attachments twice,one in message,and the other
> in Lucene Document for attachment.2,user's one query becomes 1+n query,1 for
> query message and its all attachments,n for requery the message's every
> attachments.is there any better solution?
>
>
> dazhi
>
> Thanks for any hints!!!
>
Re: how to design Lucene Document and Field to indexing and searching
email message and attachments
Posted by Erick Erickson <er...@gmail.com>.
This problem has been discussed several times, although I can't
remember the answer. So I'd recommend searching the mail archive
first.
Lucid maintains a searchable archive, see:
http://www.lucidimagination.com/About-Search
HTH
Erick
On Wed, Apr 28, 2010 at 2:35 AM, 刘庆志 <li...@gmail.com> wrote:
> hi all:
> our bussiness system generate some data,that information structrue like
> email message,one message have some attachments,so we can use email message
> to think of our data,I need index and search the message and its
> attachments,and when display hits,must display two kinds of links for every
> hit: one kind for message,and the other for attachments which match the
> query criteria,so the former kind there is only one link,but latter may be
> zero to n links.
> one design may be as:for every message design one Lucene Document,it has a
> field to record its id,let's name the field id,an other field to correspond
> all its attachments,let's name the field attachments,afert that we also
> design a Lucene Document to correspond every message's every attachment,this
> Document has a field record its message's id,let's name the field messageid,
> so when query,we can retrieval messages may be itself's cotent or its
> attachments content match the query criteria,for generating links for
> attachments which match the query criteria,we can requey again,this time we
> can query only the message's attachments by adding a query condition that
> messageid=father query's messge id.it's obviously,there are two
> disadvantages: 1,it indexes attachments twice,one in message,and the other
> in Lucene Document for attachment.2,user's one query becomes 1+n query,1 for
> query message and its all attachments,n for requery the message's every
> attachments.is there any better solution?
>
>
> dazhi
>
> Thanks for any hints!!!
>
Re: how to design Lucene Document and Field to indexing and searching email message and attachments
Posted by 刘庆志 <li...@gmail.com>.
Hoss:
Thanks for your answer,but what means for References: <t2...@mail.gmail.com>?
I make a mistake to reply other thread's message,I realize it when I visit mailing list archives through web, this is my first time to use mail list,and I'll take care of this in future.
dazhi
----- Original Message -----
From: "Chris Hostetter" <ho...@fucit.org>
To: <ja...@lucene.apache.org>
Sent: Thursday, April 29, 2010 3:04 AM
Subject: Re: how to design Lucene Document and Field to indexing and searching email message and attachments
>
> : References: <t2...@mail.gmail.com>
> : <i2...@mail.gmail.com>
> : <x2...@mail.gmail.com>
> : <2E6A89A648463A4EBF093A9062C16683018293DDFECE@SBMAILBOX1.sb.statsbibliotek
> : et.dk> <s2...@mail.gmail.com>
> : <p2...@mail.gmail.com>
> : Subject: how to design Lucene Document and Field to indexing and searching
> : email message and attachments
>
> http://people.apache.org/~hossman/#threadhijack
> Thread Hijacking on Mailing Lists
>
> When starting a new discussion on a mailing list, please do not reply to
> an existing message, instead start a fresh email. Even if you change the
> subject line of your email, other mail headers still track which thread
> you replied to and your question is "hidden" in that thread and gets less
> attention. It makes following discussions in the mailing list archives
> particularly difficult.
> See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: how to design Lucene Document and Field to indexing and searching
email message and attachments
Posted by Chris Hostetter <ho...@fucit.org>.
: References: <t2...@mail.gmail.com>
: <i2...@mail.gmail.com>
: <x2...@mail.gmail.com>
: <2E6A89A648463A4EBF093A9062C16683018293DDFECE@SBMAILBOX1.sb.statsbibliotek
: et.dk> <s2...@mail.gmail.com>
: <p2...@mail.gmail.com>
: Subject: how to design Lucene Document and Field to indexing and searching
: email message and attachments
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
how to design Lucene Document and Field to indexing and searching email message and attachments
Posted by 刘庆志 <li...@gmail.com>.
hi all:
our bussiness system generate some data,that information structrue like email message,one message have some attachments,so we can use email message to think of our data,I need index and search the message and its attachments,and when display hits,must display two kinds of links for every hit: one kind for message,and the other for attachments which match the query criteria,so the former kind there is only one link,but latter may be zero to n links.
one design may be as:for every message design one Lucene Document,it has a field to record its id,let's name the field id,an other field to correspond all its attachments,let's name the field attachments,afert that we also design a Lucene Document to correspond every message's every attachment,this Document has a field record its message's id,let's name the field messageid, so when query,we can retrieval messages may be itself's cotent or its attachments content match the query criteria,for generating links for attachments which match the query criteria,we can requey again,this time we can query only the message's attachments by adding a query condition that messageid=father query's messge id.it's obviously,there are two disadvantages: 1,it indexes attachments twice,one in message,and the other in Lucene Document for attachment.2,user's one query becomes 1+n query,1 for query message and its all attachments,n for requery the message's every attachments.is there any better solution?
dazhi
Thanks for any hints!!!
Re: Right memory for search application
Posted by Samarendra Pratap <sa...@gmail.com>.
I have got a lot of valuable information in this thread so far.
Thanks to all.
In my last mail I mentioned only two fields because others' usage was
negligible and I thought they are not important. But now after *Toke *explained
the formulae, I think sorting on those fields would also be consuming a huge
part of memory.There are 2 other sorting fields; one of which is used in
both ascending/descending sorting.
Within next couple of days (or may be a week) I'll be
1. profiling my application,
2. analyzing and tuning GC options
However, I have a few more curiosities -
1. Tom wrote:
*Have you checked that your machine is correctly identified as a server*
*and has optimized GC settings?*
*I did not understand the meaning of "correctly identified as a server" Can
you please help me understand?*
*
*
2. *Should I change the type of fields?*
** As I said in my first mail that I have 56 fields in my index, most of
them contain a numeric value or one of system defined values (e.g. gender
field can contain only "male", "female", or "unknown"). There are only 7
fields which are indexed with user defined values.
All the fields are created with *Field*
(String name, String value, Field.Store store, Field.Index index)
It would be creating all the fields as normal string fields. Is it
*always*a good idea to use specific classes (NumericField, DateTime
etc.). We do not
have space problem if that matters.
3. *Is there any advice on number of fields?*
*Somewhere on the net I read that instead of keeping different type of
values in different fields, (e.g. field1:value1, field2:value2,...) one
should practice keeping different values in single field (e.g.
field:field1_value1,
field:field2_value2,...). But I could not confirm it from anywhere else. Any
comments?*
4. Ian wrote:
*Sorting by score down to the second will use a lot of memory. Can you*
*make it less granular?*
Is it less painful sorting on two fields; first on yymmdd and then on
yymmddHHMMSS than sorting just on latter? (Naturally it should use second
field, only where required but technically ...?)
Thanks again for the invaluable support I am getting from here.
- Samar
On Wed, Apr 28, 2010 at 9:12 AM, Lance Norskog <go...@gmail.com> wrote:
> Solr's timestamp representation (TrieDateField) is tuned for space and
> speed. It has a compressed representation, and sorts with far less
> space than Strings.
>
> Also you get something called a date facet, which lets you bucketize
> facet searches by time block.
>
> On Tue, Apr 27, 2010 at 1:02 PM, Toke Eskildsen <te...@statsbiblioteket.dk>
> wrote:
> > Samarendra Pratap [samarzone@gmail.com] wrote:
> >> 1. Our default option is sort by score, however almost 8% of searches
> use
> >> sorting on a field (yyyymmddHHMMSS). This field is indexed as string
> (not as
> >> NumericField or DateField).
> >
> > Guessing that the timestamp is practically unique for each document,
> sorting by String takes up a bit more than
> > 18M * (40 bytes + 2 * "yyyymmddHHMMSS".length() bytes) ~= 1.2 GB of RAM
> as the Strings are cached. Coupled with the normal overhead of just opening
> an index of your size (500MB by your measurements?), I would have guessed
> that 3600MB would definitely be enough to open the index and do sorted
> searches.
> >
> > I realize that fiddling with production servers is dangerous, but
> connecting with JConsole and forcing a garbage collection might be
> acceptable? That should enable you to determine whether you're leaking
> memory or if it's just the JVM being greedy. I'd guess you leaking though,
> as HotSpot does not normally allocate up to the limit if it does not need
> to.
> >
> > Anyway, changing to one of the optimized fields for sorting dates should
> shave 1 GB off the memory requirement, so I'll recommend doing that no
> matter what the main cause of your memory problems is.
> >
> > Regards,
> > Toke Eskildsen
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Regards,
Samar
Re: Right memory for search application
Posted by Lance Norskog <go...@gmail.com>.
Solr's timestamp representation (TrieDateField) is tuned for space and
speed. It has a compressed representation, and sorts with far less
space than Strings.
Also you get something called a date facet, which lets you bucketize
facet searches by time block.
On Tue, Apr 27, 2010 at 1:02 PM, Toke Eskildsen <te...@statsbiblioteket.dk> wrote:
> Samarendra Pratap [samarzone@gmail.com] wrote:
>> 1. Our default option is sort by score, however almost 8% of searches use
>> sorting on a field (yyyymmddHHMMSS). This field is indexed as string (not as
>> NumericField or DateField).
>
> Guessing that the timestamp is practically unique for each document, sorting by String takes up a bit more than
> 18M * (40 bytes + 2 * "yyyymmddHHMMSS".length() bytes) ~= 1.2 GB of RAM as the Strings are cached. Coupled with the normal overhead of just opening an index of your size (500MB by your measurements?), I would have guessed that 3600MB would definitely be enough to open the index and do sorted searches.
>
> I realize that fiddling with production servers is dangerous, but connecting with JConsole and forcing a garbage collection might be acceptable? That should enable you to determine whether you're leaking memory or if it's just the JVM being greedy. I'd guess you leaking though, as HotSpot does not normally allocate up to the limit if it does not need to.
>
> Anyway, changing to one of the optimized fields for sorting dates should shave 1 GB off the memory requirement, so I'll recommend doing that no matter what the main cause of your memory problems is.
>
> Regards,
> Toke Eskildsen
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Lance Norskog
goksron@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Right memory for search application
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Samarendra Pratap [samarzone@gmail.com] wrote:
> 1. Our default option is sort by score, however almost 8% of searches use
> sorting on a field (yyyymmddHHMMSS). This field is indexed as string (not as
> NumericField or DateField).
Guessing that the timestamp is practically unique for each document, sorting by String takes up a bit more than
18M * (40 bytes + 2 * "yyyymmddHHMMSS".length() bytes) ~= 1.2 GB of RAM as the Strings are cached. Coupled with the normal overhead of just opening an index of your size (500MB by your measurements?), I would have guessed that 3600MB would definitely be enough to open the index and do sorted searches.
I realize that fiddling with production servers is dangerous, but connecting with JConsole and forcing a garbage collection might be acceptable? That should enable you to determine whether you're leaking memory or if it's just the JVM being greedy. I'd guess you leaking though, as HotSpot does not normally allocate up to the limit if it does not need to.
Anyway, changing to one of the optimized fields for sorting dates should shave 1 GB off the memory requirement, so I'll recommend doing that no matter what the main cause of your memory problems is.
Regards,
Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Right memory for search application
Posted by Ian Lea <ia...@gmail.com>.
Sorting by score down to the second will use a lot of memory. Can you
make it less granular? And I think that switching that field to a
NumericField will give you some savings - this has come up before but
I can't remember the details. I'm sure someone else will.
--
Ian.
On Tue, Apr 27, 2010 at 2:49 PM, Samarendra Pratap <sa...@gmail.com> wrote:
> Hi Ian. Thanks for the points
>
> Here are my answers -
>
> 1. Our default option is sort by score, however almost 8% of searches use
> sorting on a field (yyyymmddHHMMSS). This field is indexed as string (not as
> NumericField or DateField).
>
> 2. We are opening readers at the time of starting the application, but
> Searchers are opened for every request choosing the right readers. A few
> days back I got an answer on this very forum that reopening a searcher is
> not as big issue as reopening a reader. However we are *not closing Searcher
> *s after request is served. Should that cause memory problem? Shouldn't GC
> handle that all as the reference of Searcher becomes inaccessible?
>
> 3. I can't do a profiling on production servers but running "top" shows that
> immediately starting search server resident memory (RES) starts from 500m
> and Virtual memory (VIRT) is around 5400m. It grows to RES => 1g in 5
> minutes and 4g within half an hour while VIRT remains same.
>
> 4. Right! I'll be doing a memory profiling. I hope I'll get some hints from
> there too.
>
> Do above points (specially # 2) indicate towards anything which I can do
> besides profiling?
>
>
>
> On Tue, Apr 27, 2010 at 6:53 PM, Ian Lea <ia...@gmail.com> wrote:
>
>> There is no simple answer. However your app does sound to be using
>> rather a lot of memory for what you describe as simple searches.
>>
>> Are you using lucene sorting? That can use lots of memory. How are
>> you using/reusing searchers/readers? Having multiple ones open, or
>> failing to close old ones, will use more memory. Does memory usage
>> grow then stabilize or keep on growing?
>>
>> A memory profiler/heap dump could tell you what is really using all the
>> space.
>>
>>
>> --
>> Ian.
>>
>> On Tue, Apr 27, 2010 at 1:51 PM, Samarendra Pratap <sa...@gmail.com>
>> wrote:
>> > Hi.
>> > I am searching for some guidance on right memory options for my Search
>> > Server application. How much memory a lucene based application should be
>> > given?
>> >
>> > Till a few days back I was running my search server on java 1.4 with
>> memory
>> > options "-Xmx3600m" which was running quite fine. After upgrading the JVM
>> to
>> > *java 6* we noticed a few times that our application fell idle (perhaps
>> > hung). It was not serving the requests although the port was open. I
>> changed
>> > the "-Xmx" from 3600m to 5000m.
>> > Currently it is running OK but this created a curiosity in my mind that
>> how
>> > to find the right memory size for a lucene based application (or any java
>> > application). I believe it heavily depends on index size but how do I
>> > calculate it (at least approximately) without hit & trial?
>> >
>> > The details about my application and server are following
>> >
>> > *[ system configuration ]*
>> > # uname -a
>> > Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007
>> x86_64
>> > x86_64 x86_64 GNU/Linux
>> >
>> > *[ java version ]*
>> > # /usr/lib/jre1.6.0_20/bin/java -version
>> > java version "1.6.0_20"
>> > Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
>> > Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
>> >
>> > *[ application command line ]*
>> > # java -Xmx5000m -server -classpath
>> > lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
>> > ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer
>> conf/search.conf
>> >
>> > *[ memory information ]*
>> > *
>> > # cat /proc/meminfo
>> > MemTotal: 8156660 kB
>> > MemFree: 46924 kB
>> > Buffers: 53768 kB
>> > Cached: 3194224 kB
>> > SwapCached: 60 kB
>> > Active: 6086272 kB
>> > Inactive: 919276 kB
>> > SwapTotal: 2096472 kB
>> > SwapFree: 2095432 kB
>> > Dirty: 1148 kB
>> > Writeback: 0 kB
>> > AnonPages: 3756496 kB
>> > Mapped: 24800 kB
>> > Slab: 1058124 kB
>> > SReclaimable: 15144 kB
>> > SUnreclaim: 1042980 kB
>> > PageTables: 12232 kB
>> > NFS_Unstable: 0 kB
>> > Bounce: 0 kB
>> > CommitLimit: 6174800 kB
>> > Committed_AS: 4588900 kB
>> > VmallocTotal: 34359738367 kB
>> > VmallocUsed: 264904 kB
>> > VmallocChunk: 34359473387 kB
>> >
>> > More info*
>> > Search Server application is running on a few servers each of which
>> contains
>> > same of copy of index.
>> > Total size of index: 23 GB
>> > Total Documents in index: 18,000,000
>> > Maximum fields per index: 56 (all analyzed, 4 small fields stored, field
>> > "contents" is the largest one, created from a text of as large as 5 - 8
>> KB)
>> > Query frequency: 1 query per second per server (in peak hours)
>> > Queries are taking around 800 - 1000 ms per query
>> >
>> > *Other memory related things in Search application*
>> > There is really nothing else which will consume noticeable memory. No
>> > database connection is made. The application simply returns the IDs from
>> the
>> > index based on the query.
>> >
>> > Any help on choosing right memory option is much appreciated.
>> >
>> > --
>> > Regards,
>> > Samar
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Regards,
> Samar
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Right memory for search application
Posted by Samarendra Pratap <sa...@gmail.com>.
Hi Ian. Thanks for the points
Here are my answers -
1. Our default option is sort by score, however almost 8% of searches use
sorting on a field (yyyymmddHHMMSS). This field is indexed as string (not as
NumericField or DateField).
2. We are opening readers at the time of starting the application, but
Searchers are opened for every request choosing the right readers. A few
days back I got an answer on this very forum that reopening a searcher is
not as big issue as reopening a reader. However we are *not closing Searcher
*s after request is served. Should that cause memory problem? Shouldn't GC
handle that all as the reference of Searcher becomes inaccessible?
3. I can't do a profiling on production servers but running "top" shows that
immediately starting search server resident memory (RES) starts from 500m
and Virtual memory (VIRT) is around 5400m. It grows to RES => 1g in 5
minutes and 4g within half an hour while VIRT remains same.
4. Right! I'll be doing a memory profiling. I hope I'll get some hints from
there too.
Do above points (specially # 2) indicate towards anything which I can do
besides profiling?
On Tue, Apr 27, 2010 at 6:53 PM, Ian Lea <ia...@gmail.com> wrote:
> There is no simple answer. However your app does sound to be using
> rather a lot of memory for what you describe as simple searches.
>
> Are you using lucene sorting? That can use lots of memory. How are
> you using/reusing searchers/readers? Having multiple ones open, or
> failing to close old ones, will use more memory. Does memory usage
> grow then stabilize or keep on growing?
>
> A memory profiler/heap dump could tell you what is really using all the
> space.
>
>
> --
> Ian.
>
> On Tue, Apr 27, 2010 at 1:51 PM, Samarendra Pratap <sa...@gmail.com>
> wrote:
> > Hi.
> > I am searching for some guidance on right memory options for my Search
> > Server application. How much memory a lucene based application should be
> > given?
> >
> > Till a few days back I was running my search server on java 1.4 with
> memory
> > options "-Xmx3600m" which was running quite fine. After upgrading the JVM
> to
> > *java 6* we noticed a few times that our application fell idle (perhaps
> > hung). It was not serving the requests although the port was open. I
> changed
> > the "-Xmx" from 3600m to 5000m.
> > Currently it is running OK but this created a curiosity in my mind that
> how
> > to find the right memory size for a lucene based application (or any java
> > application). I believe it heavily depends on index size but how do I
> > calculate it (at least approximately) without hit & trial?
> >
> > The details about my application and server are following
> >
> > *[ system configuration ]*
> > # uname -a
> > Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007
> x86_64
> > x86_64 x86_64 GNU/Linux
> >
> > *[ java version ]*
> > # /usr/lib/jre1.6.0_20/bin/java -version
> > java version "1.6.0_20"
> > Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> >
> > *[ application command line ]*
> > # java -Xmx5000m -server -classpath
> > lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
> > ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer
> conf/search.conf
> >
> > *[ memory information ]*
> > *
> > # cat /proc/meminfo
> > MemTotal: 8156660 kB
> > MemFree: 46924 kB
> > Buffers: 53768 kB
> > Cached: 3194224 kB
> > SwapCached: 60 kB
> > Active: 6086272 kB
> > Inactive: 919276 kB
> > SwapTotal: 2096472 kB
> > SwapFree: 2095432 kB
> > Dirty: 1148 kB
> > Writeback: 0 kB
> > AnonPages: 3756496 kB
> > Mapped: 24800 kB
> > Slab: 1058124 kB
> > SReclaimable: 15144 kB
> > SUnreclaim: 1042980 kB
> > PageTables: 12232 kB
> > NFS_Unstable: 0 kB
> > Bounce: 0 kB
> > CommitLimit: 6174800 kB
> > Committed_AS: 4588900 kB
> > VmallocTotal: 34359738367 kB
> > VmallocUsed: 264904 kB
> > VmallocChunk: 34359473387 kB
> >
> > More info*
> > Search Server application is running on a few servers each of which
> contains
> > same of copy of index.
> > Total size of index: 23 GB
> > Total Documents in index: 18,000,000
> > Maximum fields per index: 56 (all analyzed, 4 small fields stored, field
> > "contents" is the largest one, created from a text of as large as 5 - 8
> KB)
> > Query frequency: 1 query per second per server (in peak hours)
> > Queries are taking around 800 - 1000 ms per query
> >
> > *Other memory related things in Search application*
> > There is really nothing else which will consume noticeable memory. No
> > database connection is made. The application simply returns the IDs from
> the
> > index based on the query.
> >
> > Any help on choosing right memory option is much appreciated.
> >
> > --
> > Regards,
> > Samar
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Regards,
Samar
Re: Right memory for search application
Posted by Ian Lea <ia...@gmail.com>.
There is no simple answer. However your app does sound to be using
rather a lot of memory for what you describe as simple searches.
Are you using lucene sorting? That can use lots of memory. How are
you using/reusing searchers/readers? Having multiple ones open, or
failing to close old ones, will use more memory. Does memory usage
grow then stabilize or keep on growing?
A memory profiler/heap dump could tell you what is really using all the space.
--
Ian.
On Tue, Apr 27, 2010 at 1:51 PM, Samarendra Pratap <sa...@gmail.com> wrote:
> Hi.
> I am searching for some guidance on right memory options for my Search
> Server application. How much memory a lucene based application should be
> given?
>
> Till a few days back I was running my search server on java 1.4 with memory
> options "-Xmx3600m" which was running quite fine. After upgrading the JVM to
> *java 6* we noticed a few times that our application fell idle (perhaps
> hung). It was not serving the requests although the port was open. I changed
> the "-Xmx" from 3600m to 5000m.
> Currently it is running OK but this created a curiosity in my mind that how
> to find the right memory size for a lucene based application (or any java
> application). I believe it heavily depends on index size but how do I
> calculate it (at least approximately) without hit & trial?
>
> The details about my application and server are following
>
> *[ system configuration ]*
> # uname -a
> Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007 x86_64
> x86_64 x86_64 GNU/Linux
>
> *[ java version ]*
> # /usr/lib/jre1.6.0_20/bin/java -version
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
>
> *[ application command line ]*
> # java -Xmx5000m -server -classpath
> lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
> ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer conf/search.conf
>
> *[ memory information ]*
> *
> # cat /proc/meminfo
> MemTotal: 8156660 kB
> MemFree: 46924 kB
> Buffers: 53768 kB
> Cached: 3194224 kB
> SwapCached: 60 kB
> Active: 6086272 kB
> Inactive: 919276 kB
> SwapTotal: 2096472 kB
> SwapFree: 2095432 kB
> Dirty: 1148 kB
> Writeback: 0 kB
> AnonPages: 3756496 kB
> Mapped: 24800 kB
> Slab: 1058124 kB
> SReclaimable: 15144 kB
> SUnreclaim: 1042980 kB
> PageTables: 12232 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> CommitLimit: 6174800 kB
> Committed_AS: 4588900 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 264904 kB
> VmallocChunk: 34359473387 kB
>
> More info*
> Search Server application is running on a few servers each of which contains
> same of copy of index.
> Total size of index: 23 GB
> Total Documents in index: 18,000,000
> Maximum fields per index: 56 (all analyzed, 4 small fields stored, field
> "contents" is the largest one, created from a text of as large as 5 - 8 KB)
> Query frequency: 1 query per second per server (in peak hours)
> Queries are taking around 800 - 1000 ms per query
>
> *Other memory related things in Search application*
> There is really nothing else which will consume noticeable memory. No
> database connection is made. The application simply returns the IDs from the
> index based on the query.
>
> Any help on choosing right memory option is much appreciated.
>
> --
> Regards,
> Samar
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org