Indexing documents generated from a template

[Eric Hauser <ew...@gmail.com>]

Hi,

I'm doing research on indexing some documents that are generated from
templates.  I don't have the exact statistics yet, but I'm estimating that
in the standard case 90% of the document is the same across all instances of
the document and the other 10% is dynamic (although it is certainly possible
for it to be 10/90).  Because these documents can be rather large in size
and there could potentially be millions of instances of a single document, I
don't want to put every instance of the document into the index.

What (I think) I would like to do is index the template and the dynamic
content separately, and then merge the search results afterwards.  This does
not seem too difficult; except for when a query spans both the template and
dynamic content.  Also, things like proximity queries would be difficult.
 In theory, it would seem plausible to split the terms of the original
search, search both indexes in parallel, rebuild the document, build an in
memory index of merged document, and run the original search against it.
 Most of the searches are going to be "online" meaning that a Hadoop job
probably isn't appropriate.

I am working on compiler better statistics on the standard deviation for
static to dynamic content, but in the meantime I was just curious if anyone
else has dealt with a similar scenario or had something to point me at for
additional research.  Thanks.