External File Field

[Mark <st...@gmail.com>]

We have around 10million documents that are in our index and about 10% 
of them have some extra "statistics" that are calculated on a daily 
basis which are then index and used in our function queries. This 
reindexing comes at the expense of doing multiple joins in DIH so I am 
thinking it may be faster to precompute these values and use external 
files rather than have to re-index 10% of our corpus daily. How many 
external file fields could one use before it becomes "too many"? Is this 
a valid use case or am I trying to fit a square into a circular hole?

Re: External File Field

[Yonik Seeley <yo...@lucidimagination.com>]

On Mon, Aug 1, 2011 at 11:16 AM, Mark <st...@gmail.com> wrote:
> We have around 10million documents that are in our index and about 10% of
> them have some extra "statistics" that are calculated on a daily basis which
> are then index and used in our function queries. This reindexing comes at
> the expense of doing multiple joins in DIH so I am thinking it may be faster
> to precompute these values and use external files rather than have to
> re-index 10% of our corpus daily. How many external file fields could one
> use before it becomes "too many"? Is this a valid use case or am I trying to
> fit a square into a circular hole?

Each external file field will take up maxDoc*4 bytes of RAM.
The other consideration is the time to load them (how often the index
needs to change).

-Yonik
http://www.lucidimagination.com