Block-Join and number of child-documents

[Ravikumar Govindarajan <ra...@gmail.com>]

We are planning to use block-indexing and ToChildBlockJoin queries...

Each parent-doc can contain anywhere between 1-2000 children-docs and is
highly variable.

A sample user-stats is as follows

1. No.of. parent-docs = 500K
2. Children -per parent = 50
3. Total-docs = 25 Million
4. Size occupied = 7 GB

A given index contains many such users but we are planning to limit the
size to 32GB per-index. When exceeded, addDocuments() call moves to newer
index.

The number of docs in one 32GB index sounds very scary. Are queries
affected by such high number of documents? Filters, AcceptDocs etc...
 could also become memory-heavy no?

Is Block-Join the correct fit for the above scenario?

--
Ravi

Re: Block-Join and number of child-documents

[Ravikumar Govindarajan <ra...@gmail.com>]

FYI, there is SirenDB on top of lucene that addresses such concerns...

It supports multi-level parent-child relationships and provides nice
querying capabilities...

--
Ravi


On Thu, Jul 31, 2014 at 12:59 PM, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> We are planning to use block-indexing and ToChildBlockJoin queries...
>
> Each parent-doc can contain anywhere between 1-2000 children-docs and is
> highly variable.
>
> A sample user-stats is as follows
>
> 1. No.of. parent-docs = 500K
> 2. Children -per parent = 50
> 3. Total-docs = 25 Million
> 4. Size occupied = 7 GB
>
> A given index contains many such users but we are planning to limit the
> size to 32GB per-index. When exceeded, addDocuments() call moves to newer
> index.
>
> The number of docs in one 32GB index sounds very scary. Are queries
> affected by such high number of documents? Filters, AcceptDocs etc...
>  could also become memory-heavy no?
>
> Is Block-Join the correct fit for the above scenario?
>
> --
> Ravi
>
>