You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Kevin A. Burton" <bu...@newsmonster.org> on 2003/08/11 09:07:04 UTC
2,147,483,647 max documents?
Why was an int chosen to represent document handles? Is there a reason
for this? Why wasn't a long chosen to represent document handles? 64
bits seems like the obvious choice here except for a potentially bloated
datastore.... (32 extra bits)
I guess one possible solution is to use multiple indexes. This way you
could run the search on each index and build a 64bit handle with the
first 32bits being the index handle and the second 32bits being the
local handle.
Kevin
--
Help Support NewsMonster Development! Purchase NewsMonster PRO!
http://www.newsmonster.org/download-pro.html
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
AIM - sfburtonator, Web - http://www.peerfear.org/
GPG fingerprint: 4D20 40A0 C734 307E C7B4 DCAA 0303 3AC5 BD9D 7C4D
IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: 2,147,483,647 max documents?
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Monday 11 August 2003 01:07, Kevin A. Burton wrote:
> Why was an int chosen to represent document handles? Is there a reason
> for this? Why wasn't a long chosen to represent document handles? 64
> bits seems like the obvious choice here except for a potentially bloated
> datastore.... (32 extra bits)
I can't speak for actual reasons (not being core Lucene developer), but the
general benefits of 32-bit ints vs. longs are:
- Better performance on pretty much any current architecture (even so-called
64-bit CPUs often prefer 32-bit data access, and 64-bit representations are
more important for addressing).
Also, smaller data set size is usually also good for performance (caching).
- Atomicity of access (read access can often be done without synchronizing);
longs can not be atomically accessed in Java.
Another question is whether limited address space presents a real problem.
Since Lucene can reuse doc ids (or rather, there is not persistent id per se?
doc id is just an index, and holes left by removed docs can be reused?),
perhaps this is usually not much of an issue?
-+ Tatu +-
Re: 2,147,483,647 max documents?
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Monday 11 August 2003 01:07, Kevin A. Burton wrote:
> Why was an int chosen to represent document handles? Is there a reason
> for this? Why wasn't a long chosen to represent document handles? 64
> bits seems like the obvious choice here except for a potentially bloated
> datastore.... (32 extra bits)
I can't speak for actual reasons (not being core Lucene developer), but the
general benefits of 32-bit ints vs. longs are:
- Better performance on pretty much any current architecture (even so-called
64-bit CPUs often prefer 32-bit data access, and 64-bit representations are
more important for addressing).
Also, smaller data set size is usually also good for performance (caching).
- Atomicity of access (read access can often be done without synchronizing);
longs can not be atomically accessed in Java.
Another question is whether limited address space presents a real problem.
Since Lucene can reuse doc ids (or rather, there is not persistent id per se?
doc id is just an index, and holes left by removed docs can be reused?),
perhaps this is usually not much of an issue?
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org