You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Alex Pang <pa...@gmail.com> on 2015/07/20 23:48:46 UTC

Lucene 5.2.0 global ordinal based query time join on multiple indexes

Hi,



Does the Global Ordinal based query time join support joining on multiple
indexes?



>From my testing on 2 indexes with a common join field, the document ids I
get back from the ScoreDoc[] when searching are incorrect, though the
number of results is the same as if I use the older join query.


For the parent (to) index, the value of the join field is unique to each
document.

For the child (from) index, multiple documents can have the same value for
the join field, which must be found in the parent index.

Both indexes have a join field indexed with SortedDocValuesField.


The parent index had 7 segments and child index had 3 segments.


Ordinal map is built with:

    SortedDocValues[] values = new SortedDocValues[searcher1

        .getIndexReader().leaves().size()];

    for (LeafReaderContext leadContext : searcher1.getIndexReader()

        .leaves()) {

      values[leadContext.ord] = DocValues.getSorted(leadContext.reader(),

          "join_field");

    }

    MultiDocValues.OrdinalMap ordinalMap = null;

    ordinalMap = MultiDocValues.OrdinalMap.build(searcher1.getIndexReader()

        .getCoreCacheKey(), values, PackedInts.DEFAULT);


Join Query:

joinQuery = JoinUtil.createJoinQuery("join_field",

          fromQuery,

          new TermQuery(new Term("type", "to")), searcher2,

          ScoreMode.Max, ordinalMap);



Thanks,

Alex

Re: Lucene 5.2.0 global ordinal based query time join on multiple indexes

Posted by Martijn v Groningen <ma...@gmail.com>.
Hi Alex,

I never took joining based on multiple indices into account when creating
the global ordinal join.
I like your idea of using a multi reader to make the join think it is
running on one index. But I don't know if there are edge cases where this
might not work.

Martijn

On 21 July 2015 at 21:12, Alex Pang <pa...@gmail.com> wrote:

> Seems if I create a MultiReader from my index searchers and create the
> ordinal map from that MultiReader (and use an IndexSearcher created from
> the MultiReader in the createJoinQuery), then the correct results are
> found.
>
>
> On Mon, Jul 20, 2015 at 5:48 PM, Alex Pang <pa...@gmail.com> wrote:
>
> > Hi,
> >
> >
> >
> > Does the Global Ordinal based query time join support joining on multiple
> > indexes?
> >
> >
> >
> > From my testing on 2 indexes with a common join field, the document ids I
> > get back from the ScoreDoc[] when searching are incorrect, though the
> > number of results is the same as if I use the older join query.
> >
> >
> > For the parent (to) index, the value of the join field is unique to each
> > document.
> >
> > For the child (from) index, multiple documents can have the same value
> for
> > the join field, which must be found in the parent index.
> >
> > Both indexes have a join field indexed with SortedDocValuesField.
> >
> >
> > The parent index had 7 segments and child index had 3 segments.
> >
> >
> > Ordinal map is built with:
> >
> >     SortedDocValues[] values = new SortedDocValues[searcher1
> >
> >         .getIndexReader().leaves().size()];
> >
> >     for (LeafReaderContext leadContext : searcher1.getIndexReader()
> >
> >         .leaves()) {
> >
> >       values[leadContext.ord] = DocValues.getSorted(leadContext.reader(),
> >
> >           "join_field");
> >
> >     }
> >
> >     MultiDocValues.OrdinalMap ordinalMap = null;
> >
> >     ordinalMap =
> MultiDocValues.OrdinalMap.build(searcher1.getIndexReader()
> >
> >         .getCoreCacheKey(), values, PackedInts.DEFAULT);
> >
> >
> > Join Query:
> >
> > joinQuery = JoinUtil.createJoinQuery("join_field",
> >
> >           fromQuery,
> >
> >           new TermQuery(new Term("type", "to")), searcher2,
> >
> >           ScoreMode.Max, ordinalMap);
> >
> >
> >
> > Thanks,
> >
> > Alex
> >
>



-- 
Met vriendelijke groet,

Martijn van Groningen

Re: Lucene 5.2.0 global ordinal based query time join on multiple indexes

Posted by Alex Pang <pa...@gmail.com>.
Seems if I create a MultiReader from my index searchers and create the
ordinal map from that MultiReader (and use an IndexSearcher created from
the MultiReader in the createJoinQuery), then the correct results are found.


On Mon, Jul 20, 2015 at 5:48 PM, Alex Pang <pa...@gmail.com> wrote:

> Hi,
>
>
>
> Does the Global Ordinal based query time join support joining on multiple
> indexes?
>
>
>
> From my testing on 2 indexes with a common join field, the document ids I
> get back from the ScoreDoc[] when searching are incorrect, though the
> number of results is the same as if I use the older join query.
>
>
> For the parent (to) index, the value of the join field is unique to each
> document.
>
> For the child (from) index, multiple documents can have the same value for
> the join field, which must be found in the parent index.
>
> Both indexes have a join field indexed with SortedDocValuesField.
>
>
> The parent index had 7 segments and child index had 3 segments.
>
>
> Ordinal map is built with:
>
>     SortedDocValues[] values = new SortedDocValues[searcher1
>
>         .getIndexReader().leaves().size()];
>
>     for (LeafReaderContext leadContext : searcher1.getIndexReader()
>
>         .leaves()) {
>
>       values[leadContext.ord] = DocValues.getSorted(leadContext.reader(),
>
>           "join_field");
>
>     }
>
>     MultiDocValues.OrdinalMap ordinalMap = null;
>
>     ordinalMap = MultiDocValues.OrdinalMap.build(searcher1.getIndexReader()
>
>         .getCoreCacheKey(), values, PackedInts.DEFAULT);
>
>
> Join Query:
>
> joinQuery = JoinUtil.createJoinQuery("join_field",
>
>           fromQuery,
>
>           new TermQuery(new Term("type", "to")), searcher2,
>
>           ScoreMode.Max, ordinalMap);
>
>
>
> Thanks,
>
> Alex
>