You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "McKinley, James T" <ja...@cengage.com> on 2015/01/21 16:00:22 UTC
ToChildBlockJoinQuery question
Hi,
I'm attempting to use ToChildBlockJoinQuery in Lucene 4.8.1 by following Mike McCandless' blog post:
http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html
I have a set of child documents which are named works and a set of parent documents which are named persons that are the creators of the named works. The parent document has a nationality and the child document does not. I want to query the children (named works) limiting by the nationality of the parent (named person). I've indexed the documents as follows (I'm pulling the docs from an existing index):
private void createNamedWorkIndex(String srcIndexPath, String destIndexPath) throws IOException {
FSDirectory srcDir = FSDirectory.open(new File(srcIndexPath));
FSDirectory destDir = FSDirectory.open(new File(destIndexPath));
IndexReader reader = DirectoryReader.open(srcDir);
Version version = Version.LUCENE_48;
IndexWriterConfig conf = new IndexWriterConfig(version, new StandardTextAnalyzer(version));
Set<String> crids = getCreatorIds(reader);
String[] crida = crids.toArray(new String[crids.size()]);
int numThreads = 24;
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
int numCrids = crids.size();
int batchSize = numCrids / numThreads;
int remainder = numCrids % numThreads;
System.out.println("Inserting work/creator blocks using " + numThreads + " threads...");
try (IndexWriter writer = new IndexWriter(destDir, conf)){
for (int i = 0; i < numThreads; i++) {
String[] cridRange;
if (i == numThreads - 1) {
cridRange = Arrays.copyOfRange(crida, i*batchSize, ((i+1)*batchSize - 1) + remainder);
} else {
cridRange = Arrays.copyOfRange(crida, i*batchSize, ((i+1)*batchSize - 1));
}
String id = "" + ((char)('A' + i));
Runnable indexer = new IndexRunnable(id , reader, writer, new HashSet<String>(Arrays.asList(cridRange)));
executor.execute(indexer);
}
executor.shutdown();
executor.awaitTermination(2, TimeUnit.HOURS);
} catch (Exception e) {
executor.shutdownNow();
throw new RuntimeException(e);
} finally {
reader.close();
srcDir.close();
destDir.close();
}
System.out.println("Done!");
}
public static class IndexRunnable implements Runnable {
private String id;
private IndexReader reader;
private IndexWriter writer;
private Set<String> crids;
public IndexRunnable(String id, IndexReader reader, IndexWriter writer, Set<String> crids) {
this.id = id;
this.reader = reader;
this.writer = writer;
this.crids = crids;
}
@Override
public void run() {
IndexSearcher searcher = new IndexSearcher(reader);
try {
int count = 0;
for (String crid : crids) {
List<Document> docs = new ArrayList<>();
BooleanQuery abidQuery = new BooleanQuery();
abidQuery.add(new TermQuery(new Term("ABID", crid)), Occur.MUST);
abidQuery.add(new TermQuery(new Term("AGPR", "true")), Occur.MUST);
TermQuery cridQuery = new TermQuery(new Term("CRID", crid));
TopDocs creatorDocs = searcher.search(abidQuery, Integer.MAX_VALUE);
TopDocs workDocs = searcher.search(cridQuery, Integer.MAX_VALUE);
for (int i = 0; i < workDocs.scoreDocs.length; i++) {
docs.add(reader.document(workDocs.scoreDocs[i].doc));
}
if (creatorDocs.scoreDocs.length > 0) {
docs.add(reader.document(creatorDocs.scoreDocs[0].doc));
}
writer.addDocuments(docs);
if (++count % 100 == 0) {
System.out.println(id + " = " + count);
writer.commit();
}
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
I then attempt to perform a block join query as follows:
private void runToChildBlockJoinQuery(String indexPath) throws IOException {
FSDirectory dir = FSDirectory.open(new File(indexPath));
IndexReader reader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(reader);
TermQuery parentQuery = new TermQuery(new Term("NT", "american"));
TermQuery parentFilterQuery = new TermQuery(new Term("AGTY", "np"));
Filter parentFilter = new CachingWrapperFilter(new QueryWrapperFilter(parentFilterQuery));
ToChildBlockJoinQuery tcbjq = new ToChildBlockJoinQuery(parentQuery, parentFilter, true);
TopDocs worksDocs = searcher.search(tcbjq, 20);
displayWorks(reader, searcher, worksDocs);
}
and I get the following exception:
Exception in thread "main" java.lang.IllegalStateException: parentFilter must return FixedBitSet; got org.apache.lucene.util.WAH8DocIdSet@34e671de
at org.apache.lucene.search.join.ToChildBlockJoinQuery$ToChildBlockJoinWeight.scorer(ToChildBlockJoinQuery.java:148)
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
at BlockJoinQueryTester.runToChildBlockJoinQuery(BlockJoinQueryTester.java:73)
at BlockJoinQueryTester.main(BlockJoinQueryTester.java:40)
I don't understand what I'm doing wrong and what a "FixedBitSet" is and why I don't get one out of my filter. Is FixedBitSet a special kind of OpenBitSet and what does "fixed" mean in this context? Thanks for any help.
Jim
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ToChildBlockJoinQuery question
Posted by Michael Sokolov <ms...@safaribooksonline.com>.
Great! Thanks for letting us know
-Mike
On 1/22/15 2:07 PM, McKinley, James T wrote:
> Hi Mike,
>
> I guess given the difficulty I've had getting the block join query to work it didn't occur to me to try and combine it in a BooleanQuery. :P Using the BJQ in a BooleanQuery with other TermQuerys works fine and does exactly what I wanted! Thanks very much for your help!
>
> Jim
> ________________________________________
> From: Michael Sokolov [msokolov@safaribooksonline.com]
> Sent: Thursday, January 22, 2015 11:45 AM
> To: java-user@lucene.apache.org
> Subject: Re: ToChildBlockJoinQuery question
>
> I think the idea is that you create a blockjoinquery that encapsulates
> the join relation, and then you can create additional constraints in the
> result document space. In the case of ToChildBJQ, the result documents
> are child documents, so any additional query constraints will be applied
> to child documents. For example, you could create the
>
> ToChildBlockJoinQuery bjq = jamesBJQ();
> TermQuery tq = new TermQuery (new Term("title", "doctor"));
> BooleanQuery bq = new BooleanQuery (bjq, tq);
>
> bq would then match books with parent (ie author) restrictions defined
> in jamesBJQ(), and child (ie book) restrictions defined by other queries
> like tq (title:doctor)
>
> -Mike
>
> On 1/22/15 11:27 AM, McKinley, James T wrote:
>> Hi Greg,
>>
>> Thanks describing how block join queries were intended to work. Your description makes sense to me, however according to the API docs:
>>
>> http://lucene.apache.org/core/4_8_0/join/org/apache/lucene/search/join/ToChildBlockJoinQuery.html
>>
>> and particularly the naming of the parameters I don't think the API actually works as you described:
>>
>> ToChildBlockJoinQuery(Query parentQuery, Filter parentsFilter, boolean doScores)
>>
>> If the filter was intended to filter the child docs I think it would be called childFilter no?
>>
>> I think the use of the CachingWrappingFilter in the example I got from Mike McCandless' blog post was the real cause of the exception I was seeing (maybe things have changed internally since that post). I finally noticed a mention of the FixedBitSetCachingWrapperFilter in the description of the ToChildBlockJoinQuery constructor in the API docs. When I changed to using a filter produced by the FixedBitSetCachingWrapperFilter class the IllegalStateException no longer occurs and I get the child docs using ToChildBlockJoinQuery with a parent doc filter and parent doc query and results look correctly limited by the parent constraints. For example:
>>
>> ...
>> Gub-Gub's Book: An Encyclopedia of Food (Fictional work), Fictional work, 119320101
>> by: Lofting, Hugh - NP, American, Writer
>>
>> The Story of Doctor Dolittle, Being the History of His Peculiar Life at Home and Astonishing Adventures in Foreign Parts (Novel), Novel, 119200101
>> by: Lofting, Hugh - NP, American, Writer
>>
>> The Voyages of Doctor Dolittle (Novel), Novel, 119220101
>> by: Lofting, Hugh - NP, American, Writer
>>
>> The Story of Doctor Dolittle (Novel), Novel, 119200101
>> by: Lofting, Hugh - NP, American, Writer
>>
>> ...
>> Mister Beers (Poem), Poem, null
>> by: Lofting, Hugh - NP, American, Writer
>>
>> The Twilight of Magic (Novel), Novel, 119300101
>> by: Lofting, Hugh - NP, American, Writer
>>
>> Picnic (Lofting, Hugh) (Poem), Poem, null
>> by: Lofting, Hugh - NP, American, Writer
>>
>> The Impossible Patriotism Project (Picture story), Picture story, 120070101
>>
>> A Skeleton in God's Closet: A Novel (Novel), Novel, 119940101
>> by: Maier, Paul Luther - NP, American, null
>>
>> Pontius Pilate (Novel), Novel, 119680101
>> by: Maier, Paul Luther - NP, American, null
>>
>> ...
>> Josephus: The Essential Writings (Collection), Collection, 119880101
>> by: Maier, Paul Luther - NP, American, null
>>
>> She Said the Geese (Poem), Poem, null
>> by: Lifshin, Lyn - NP, American, Poet
>>
>> She Said She Could See Music (Poem), Poem, null
>> by: Lifshin, Lyn - NP, American, Poet
>> ...
>>
>> However I see no way to further limit the children as you describe. If I use "a query that matches the set of parents and a filter that matches the set of children" as you suggest I get no results back. I think your description of how it should work makes complete sense, but that is not what I'm seeing when I try it. Here's the code that produced the above output:
>>
>> private void runToChildBlockJoinQuery(String indexPath) throws IOException {
>> FSDirectory dir = FSDirectory.open(new File(indexPath));
>> IndexReader reader = DirectoryReader.open(dir);
>> IndexSearcher searcher = new IndexSearcher(reader);
>>
>> TermQuery parentFilterQuery = new TermQuery(new Term("AGTY", "np"));
>> BooleanQuery parentQuery = new BooleanQuery();
>> parentQuery.add(new TermQuery(new Term("AGTY", "np")), Occur.MUST);
>> parentQuery.add(new TermQuery(new Term("NT", "american")), Occur.MUST);
>>
>> Filter parentFilter = new FixedBitSetCachingWrapperFilter(new QueryWrapperFilter(parentFilterQuery));
>>
>> ToChildBlockJoinQuery tcbjq = new ToChildBlockJoinQuery(parentQuery, parentFilter, true);
>>
>> TopDocs worksDocs = searcher.search(tcbjq, 5000);
>>
>> System.out.println("\n*ToChildBlockJoinQuery hit count = " + worksDocs.scoreDocs.length);
>> displayWorks(reader, searcher, worksDocs);
>> }
>>
>> private void displayWorks(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
>> for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
>> String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
>> String tw = reader.document(worksDocs.scoreDocs[i].doc).get("TW");
>> String pd = reader.document(worksDocs.scoreDocs[i].doc).get("PD");
>> String crid = reader.document(worksDocs.scoreDocs[i].doc).get("CRID");
>> TopDocs creatorDocs = searcher.search(new TermQuery(new Term("ABID", crid)), Integer.MAX_VALUE);
>> System.out.println("\n" + agdn + ", " + tw + ", " + pd);
>> displayCreators(reader, searcher, creatorDocs);
>> }
>> }
>>
>> private void displayCreators(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
>> for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
>> String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
>> String agty = reader.document(worksDocs.scoreDocs[i].doc).get("AGTY");
>> String nt = reader.document(worksDocs.scoreDocs[i].doc).get("NT");
>> String poc = reader.document(worksDocs.scoreDocs[i].doc).get("POC");
>> System.out.println("\tby: " + agdn + " - " + agty + ", " +nt + ", " + poc);
>> }
>> }
>>
>> When I try to use ToParentBlockJoinQuery I don't get any results either and it is not what I really want anyway, I want the child documents limited by the parent documents.
>>
>> ToChildBlockJoinQuery almost gives me what I want, but I really need to be able to filter the child docs returned as well as the parent from which they came. If you (or anybody) still thinks I'm doing it wrong please let me know. If I should file a bug report also let me know that, I have a small index I can provide if it is useful. Thanks again for your help.
>>
>> Jim
>>
>> ________________________________________
>> From: Gregory Dearing [gregdearing@gmail.com]
>> Sent: Wednesday, January 21, 2015 6:59 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: ToChildBlockJoinQuery question
>>
>> Jim,
>>
>> I think you hit the nail on the head... that's not what BlockJoinQueries do.
>>
>> If you're wanting to search for children and join to their parents... then
>> use ToParentBlockJoinQuery, with a query that matches the set of children
>> and a filter that matches the set of parents.
>>
>> If you're searching for parents, then joining to their children... then use
>> ToChildBlockJoinQuery, with a query that matches the set of parents and a
>> filter that matches the set of children.
>>
>> When you add related documents to the index (via addDocuments), make that
>> children are added before their parents.
>>
>> The reason all the above is necessary is that it makes it possible to have
>> a nested hierarchy of relationships (ie. Parents have Children, which have
>> Children of their own). You need a query to indicate which part of the
>> hierarchy you're starting from, and a filter indicating which part of the
>> hierarchy you're joining to.
>>
>> Also, you will always get an exception if your query and your filter both
>> match the same document. A child can't be its own parent.
>>
>> BlockJoin is a very powerful feature, but what it's really doing is
>> modelling relationships using an index that doesn't know what a
>> relationship is. The relationships are determined by a combination of the
>> order that you indexed the block, and the format of your query. This
>> disjoin can lead to some weird behavior if you're not absolutely sure how
>> it works.
>>
>> Thanks,
>> Greg
>>
>>
>>
>>
>>
>> On Wed, Jan 21, 2015 at 4:34 PM, McKinley, James T <
>> james.mckinley@cengage.com> wrote:
>>
>>> Am I understanding how this is supposed to work? What I think I am (and
>>> should be) doing is providing a query and filter that specifies the parent
>>> docs and the ToChildBlockJoinQuery should return me all the child docs for
>>> the resulting parent docs. Is this correct? The reason I think I'm not
>>> understanding is that I don't see why I need both a filter and a query to
>>> specify the parent docs when a single query or filter should suffice. Am I
>>> misunderstanding what parentQuery and parentFilter mean, they both refer to
>>> parent docs right?
>>>
>>> Jim
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: ToChildBlockJoinQuery question
Posted by "McKinley, James T" <ja...@cengage.com>.
Hi Mike,
I guess given the difficulty I've had getting the block join query to work it didn't occur to me to try and combine it in a BooleanQuery. :P Using the BJQ in a BooleanQuery with other TermQuerys works fine and does exactly what I wanted! Thanks very much for your help!
Jim
________________________________________
From: Michael Sokolov [msokolov@safaribooksonline.com]
Sent: Thursday, January 22, 2015 11:45 AM
To: java-user@lucene.apache.org
Subject: Re: ToChildBlockJoinQuery question
I think the idea is that you create a blockjoinquery that encapsulates
the join relation, and then you can create additional constraints in the
result document space. In the case of ToChildBJQ, the result documents
are child documents, so any additional query constraints will be applied
to child documents. For example, you could create the
ToChildBlockJoinQuery bjq = jamesBJQ();
TermQuery tq = new TermQuery (new Term("title", "doctor"));
BooleanQuery bq = new BooleanQuery (bjq, tq);
bq would then match books with parent (ie author) restrictions defined
in jamesBJQ(), and child (ie book) restrictions defined by other queries
like tq (title:doctor)
-Mike
On 1/22/15 11:27 AM, McKinley, James T wrote:
> Hi Greg,
>
> Thanks describing how block join queries were intended to work. Your description makes sense to me, however according to the API docs:
>
> http://lucene.apache.org/core/4_8_0/join/org/apache/lucene/search/join/ToChildBlockJoinQuery.html
>
> and particularly the naming of the parameters I don't think the API actually works as you described:
>
> ToChildBlockJoinQuery(Query parentQuery, Filter parentsFilter, boolean doScores)
>
> If the filter was intended to filter the child docs I think it would be called childFilter no?
>
> I think the use of the CachingWrappingFilter in the example I got from Mike McCandless' blog post was the real cause of the exception I was seeing (maybe things have changed internally since that post). I finally noticed a mention of the FixedBitSetCachingWrapperFilter in the description of the ToChildBlockJoinQuery constructor in the API docs. When I changed to using a filter produced by the FixedBitSetCachingWrapperFilter class the IllegalStateException no longer occurs and I get the child docs using ToChildBlockJoinQuery with a parent doc filter and parent doc query and results look correctly limited by the parent constraints. For example:
>
> ...
> Gub-Gub's Book: An Encyclopedia of Food (Fictional work), Fictional work, 119320101
> by: Lofting, Hugh - NP, American, Writer
>
> The Story of Doctor Dolittle, Being the History of His Peculiar Life at Home and Astonishing Adventures in Foreign Parts (Novel), Novel, 119200101
> by: Lofting, Hugh - NP, American, Writer
>
> The Voyages of Doctor Dolittle (Novel), Novel, 119220101
> by: Lofting, Hugh - NP, American, Writer
>
> The Story of Doctor Dolittle (Novel), Novel, 119200101
> by: Lofting, Hugh - NP, American, Writer
>
> ...
> Mister Beers (Poem), Poem, null
> by: Lofting, Hugh - NP, American, Writer
>
> The Twilight of Magic (Novel), Novel, 119300101
> by: Lofting, Hugh - NP, American, Writer
>
> Picnic (Lofting, Hugh) (Poem), Poem, null
> by: Lofting, Hugh - NP, American, Writer
>
> The Impossible Patriotism Project (Picture story), Picture story, 120070101
>
> A Skeleton in God's Closet: A Novel (Novel), Novel, 119940101
> by: Maier, Paul Luther - NP, American, null
>
> Pontius Pilate (Novel), Novel, 119680101
> by: Maier, Paul Luther - NP, American, null
>
> ...
> Josephus: The Essential Writings (Collection), Collection, 119880101
> by: Maier, Paul Luther - NP, American, null
>
> She Said the Geese (Poem), Poem, null
> by: Lifshin, Lyn - NP, American, Poet
>
> She Said She Could See Music (Poem), Poem, null
> by: Lifshin, Lyn - NP, American, Poet
> ...
>
> However I see no way to further limit the children as you describe. If I use "a query that matches the set of parents and a filter that matches the set of children" as you suggest I get no results back. I think your description of how it should work makes complete sense, but that is not what I'm seeing when I try it. Here's the code that produced the above output:
>
> private void runToChildBlockJoinQuery(String indexPath) throws IOException {
> FSDirectory dir = FSDirectory.open(new File(indexPath));
> IndexReader reader = DirectoryReader.open(dir);
> IndexSearcher searcher = new IndexSearcher(reader);
>
> TermQuery parentFilterQuery = new TermQuery(new Term("AGTY", "np"));
> BooleanQuery parentQuery = new BooleanQuery();
> parentQuery.add(new TermQuery(new Term("AGTY", "np")), Occur.MUST);
> parentQuery.add(new TermQuery(new Term("NT", "american")), Occur.MUST);
>
> Filter parentFilter = new FixedBitSetCachingWrapperFilter(new QueryWrapperFilter(parentFilterQuery));
>
> ToChildBlockJoinQuery tcbjq = new ToChildBlockJoinQuery(parentQuery, parentFilter, true);
>
> TopDocs worksDocs = searcher.search(tcbjq, 5000);
>
> System.out.println("\n*ToChildBlockJoinQuery hit count = " + worksDocs.scoreDocs.length);
> displayWorks(reader, searcher, worksDocs);
> }
>
> private void displayWorks(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
> for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
> String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
> String tw = reader.document(worksDocs.scoreDocs[i].doc).get("TW");
> String pd = reader.document(worksDocs.scoreDocs[i].doc).get("PD");
> String crid = reader.document(worksDocs.scoreDocs[i].doc).get("CRID");
> TopDocs creatorDocs = searcher.search(new TermQuery(new Term("ABID", crid)), Integer.MAX_VALUE);
> System.out.println("\n" + agdn + ", " + tw + ", " + pd);
> displayCreators(reader, searcher, creatorDocs);
> }
> }
>
> private void displayCreators(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
> for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
> String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
> String agty = reader.document(worksDocs.scoreDocs[i].doc).get("AGTY");
> String nt = reader.document(worksDocs.scoreDocs[i].doc).get("NT");
> String poc = reader.document(worksDocs.scoreDocs[i].doc).get("POC");
> System.out.println("\tby: " + agdn + " - " + agty + ", " +nt + ", " + poc);
> }
> }
>
> When I try to use ToParentBlockJoinQuery I don't get any results either and it is not what I really want anyway, I want the child documents limited by the parent documents.
>
> ToChildBlockJoinQuery almost gives me what I want, but I really need to be able to filter the child docs returned as well as the parent from which they came. If you (or anybody) still thinks I'm doing it wrong please let me know. If I should file a bug report also let me know that, I have a small index I can provide if it is useful. Thanks again for your help.
>
> Jim
>
> ________________________________________
> From: Gregory Dearing [gregdearing@gmail.com]
> Sent: Wednesday, January 21, 2015 6:59 PM
> To: java-user@lucene.apache.org
> Subject: Re: ToChildBlockJoinQuery question
>
> Jim,
>
> I think you hit the nail on the head... that's not what BlockJoinQueries do.
>
> If you're wanting to search for children and join to their parents... then
> use ToParentBlockJoinQuery, with a query that matches the set of children
> and a filter that matches the set of parents.
>
> If you're searching for parents, then joining to their children... then use
> ToChildBlockJoinQuery, with a query that matches the set of parents and a
> filter that matches the set of children.
>
> When you add related documents to the index (via addDocuments), make that
> children are added before their parents.
>
> The reason all the above is necessary is that it makes it possible to have
> a nested hierarchy of relationships (ie. Parents have Children, which have
> Children of their own). You need a query to indicate which part of the
> hierarchy you're starting from, and a filter indicating which part of the
> hierarchy you're joining to.
>
> Also, you will always get an exception if your query and your filter both
> match the same document. A child can't be its own parent.
>
> BlockJoin is a very powerful feature, but what it's really doing is
> modelling relationships using an index that doesn't know what a
> relationship is. The relationships are determined by a combination of the
> order that you indexed the block, and the format of your query. This
> disjoin can lead to some weird behavior if you're not absolutely sure how
> it works.
>
> Thanks,
> Greg
>
>
>
>
>
> On Wed, Jan 21, 2015 at 4:34 PM, McKinley, James T <
> james.mckinley@cengage.com> wrote:
>
>> Am I understanding how this is supposed to work? What I think I am (and
>> should be) doing is providing a query and filter that specifies the parent
>> docs and the ToChildBlockJoinQuery should return me all the child docs for
>> the resulting parent docs. Is this correct? The reason I think I'm not
>> understanding is that I don't see why I need both a filter and a query to
>> specify the parent docs when a single query or filter should suffice. Am I
>> misunderstanding what parentQuery and parentFilter mean, they both refer to
>> parent docs right?
>>
>> Jim
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ToChildBlockJoinQuery question
Posted by Michael Sokolov <ms...@safaribooksonline.com>.
I think the idea is that you create a blockjoinquery that encapsulates
the join relation, and then you can create additional constraints in the
result document space. In the case of ToChildBJQ, the result documents
are child documents, so any additional query constraints will be applied
to child documents. For example, you could create the
ToChildBlockJoinQuery bjq = jamesBJQ();
TermQuery tq = new TermQuery (new Term("title", "doctor"));
BooleanQuery bq = new BooleanQuery (bjq, tq);
bq would then match books with parent (ie author) restrictions defined
in jamesBJQ(), and child (ie book) restrictions defined by other queries
like tq (title:doctor)
-Mike
On 1/22/15 11:27 AM, McKinley, James T wrote:
> Hi Greg,
>
> Thanks describing how block join queries were intended to work. Your description makes sense to me, however according to the API docs:
>
> http://lucene.apache.org/core/4_8_0/join/org/apache/lucene/search/join/ToChildBlockJoinQuery.html
>
> and particularly the naming of the parameters I don't think the API actually works as you described:
>
> ToChildBlockJoinQuery(Query parentQuery, Filter parentsFilter, boolean doScores)
>
> If the filter was intended to filter the child docs I think it would be called childFilter no?
>
> I think the use of the CachingWrappingFilter in the example I got from Mike McCandless' blog post was the real cause of the exception I was seeing (maybe things have changed internally since that post). I finally noticed a mention of the FixedBitSetCachingWrapperFilter in the description of the ToChildBlockJoinQuery constructor in the API docs. When I changed to using a filter produced by the FixedBitSetCachingWrapperFilter class the IllegalStateException no longer occurs and I get the child docs using ToChildBlockJoinQuery with a parent doc filter and parent doc query and results look correctly limited by the parent constraints. For example:
>
> ...
> Gub-Gub's Book: An Encyclopedia of Food (Fictional work), Fictional work, 119320101
> by: Lofting, Hugh - NP, American, Writer
>
> The Story of Doctor Dolittle, Being the History of His Peculiar Life at Home and Astonishing Adventures in Foreign Parts (Novel), Novel, 119200101
> by: Lofting, Hugh - NP, American, Writer
>
> The Voyages of Doctor Dolittle (Novel), Novel, 119220101
> by: Lofting, Hugh - NP, American, Writer
>
> The Story of Doctor Dolittle (Novel), Novel, 119200101
> by: Lofting, Hugh - NP, American, Writer
>
> ...
> Mister Beers (Poem), Poem, null
> by: Lofting, Hugh - NP, American, Writer
>
> The Twilight of Magic (Novel), Novel, 119300101
> by: Lofting, Hugh - NP, American, Writer
>
> Picnic (Lofting, Hugh) (Poem), Poem, null
> by: Lofting, Hugh - NP, American, Writer
>
> The Impossible Patriotism Project (Picture story), Picture story, 120070101
>
> A Skeleton in God's Closet: A Novel (Novel), Novel, 119940101
> by: Maier, Paul Luther - NP, American, null
>
> Pontius Pilate (Novel), Novel, 119680101
> by: Maier, Paul Luther - NP, American, null
>
> ...
> Josephus: The Essential Writings (Collection), Collection, 119880101
> by: Maier, Paul Luther - NP, American, null
>
> She Said the Geese (Poem), Poem, null
> by: Lifshin, Lyn - NP, American, Poet
>
> She Said She Could See Music (Poem), Poem, null
> by: Lifshin, Lyn - NP, American, Poet
> ...
>
> However I see no way to further limit the children as you describe. If I use "a query that matches the set of parents and a filter that matches the set of children" as you suggest I get no results back. I think your description of how it should work makes complete sense, but that is not what I'm seeing when I try it. Here's the code that produced the above output:
>
> private void runToChildBlockJoinQuery(String indexPath) throws IOException {
> FSDirectory dir = FSDirectory.open(new File(indexPath));
> IndexReader reader = DirectoryReader.open(dir);
> IndexSearcher searcher = new IndexSearcher(reader);
>
> TermQuery parentFilterQuery = new TermQuery(new Term("AGTY", "np"));
> BooleanQuery parentQuery = new BooleanQuery();
> parentQuery.add(new TermQuery(new Term("AGTY", "np")), Occur.MUST);
> parentQuery.add(new TermQuery(new Term("NT", "american")), Occur.MUST);
>
> Filter parentFilter = new FixedBitSetCachingWrapperFilter(new QueryWrapperFilter(parentFilterQuery));
>
> ToChildBlockJoinQuery tcbjq = new ToChildBlockJoinQuery(parentQuery, parentFilter, true);
>
> TopDocs worksDocs = searcher.search(tcbjq, 5000);
>
> System.out.println("\n*ToChildBlockJoinQuery hit count = " + worksDocs.scoreDocs.length);
> displayWorks(reader, searcher, worksDocs);
> }
>
> private void displayWorks(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
> for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
> String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
> String tw = reader.document(worksDocs.scoreDocs[i].doc).get("TW");
> String pd = reader.document(worksDocs.scoreDocs[i].doc).get("PD");
> String crid = reader.document(worksDocs.scoreDocs[i].doc).get("CRID");
> TopDocs creatorDocs = searcher.search(new TermQuery(new Term("ABID", crid)), Integer.MAX_VALUE);
> System.out.println("\n" + agdn + ", " + tw + ", " + pd);
> displayCreators(reader, searcher, creatorDocs);
> }
> }
>
> private void displayCreators(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
> for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
> String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
> String agty = reader.document(worksDocs.scoreDocs[i].doc).get("AGTY");
> String nt = reader.document(worksDocs.scoreDocs[i].doc).get("NT");
> String poc = reader.document(worksDocs.scoreDocs[i].doc).get("POC");
> System.out.println("\tby: " + agdn + " - " + agty + ", " +nt + ", " + poc);
> }
> }
>
> When I try to use ToParentBlockJoinQuery I don't get any results either and it is not what I really want anyway, I want the child documents limited by the parent documents.
>
> ToChildBlockJoinQuery almost gives me what I want, but I really need to be able to filter the child docs returned as well as the parent from which they came. If you (or anybody) still thinks I'm doing it wrong please let me know. If I should file a bug report also let me know that, I have a small index I can provide if it is useful. Thanks again for your help.
>
> Jim
>
> ________________________________________
> From: Gregory Dearing [gregdearing@gmail.com]
> Sent: Wednesday, January 21, 2015 6:59 PM
> To: java-user@lucene.apache.org
> Subject: Re: ToChildBlockJoinQuery question
>
> Jim,
>
> I think you hit the nail on the head... that's not what BlockJoinQueries do.
>
> If you're wanting to search for children and join to their parents... then
> use ToParentBlockJoinQuery, with a query that matches the set of children
> and a filter that matches the set of parents.
>
> If you're searching for parents, then joining to their children... then use
> ToChildBlockJoinQuery, with a query that matches the set of parents and a
> filter that matches the set of children.
>
> When you add related documents to the index (via addDocuments), make that
> children are added before their parents.
>
> The reason all the above is necessary is that it makes it possible to have
> a nested hierarchy of relationships (ie. Parents have Children, which have
> Children of their own). You need a query to indicate which part of the
> hierarchy you're starting from, and a filter indicating which part of the
> hierarchy you're joining to.
>
> Also, you will always get an exception if your query and your filter both
> match the same document. A child can't be its own parent.
>
> BlockJoin is a very powerful feature, but what it's really doing is
> modelling relationships using an index that doesn't know what a
> relationship is. The relationships are determined by a combination of the
> order that you indexed the block, and the format of your query. This
> disjoin can lead to some weird behavior if you're not absolutely sure how
> it works.
>
> Thanks,
> Greg
>
>
>
>
>
> On Wed, Jan 21, 2015 at 4:34 PM, McKinley, James T <
> james.mckinley@cengage.com> wrote:
>
>> Am I understanding how this is supposed to work? What I think I am (and
>> should be) doing is providing a query and filter that specifies the parent
>> docs and the ToChildBlockJoinQuery should return me all the child docs for
>> the resulting parent docs. Is this correct? The reason I think I'm not
>> understanding is that I don't see why I need both a filter and a query to
>> specify the parent docs when a single query or filter should suffice. Am I
>> misunderstanding what parentQuery and parentFilter mean, they both refer to
>> parent docs right?
>>
>> Jim
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: ToChildBlockJoinQuery question
Posted by "McKinley, James T" <ja...@cengage.com>.
Hi Greg,
Thanks describing how block join queries were intended to work. Your description makes sense to me, however according to the API docs:
http://lucene.apache.org/core/4_8_0/join/org/apache/lucene/search/join/ToChildBlockJoinQuery.html
and particularly the naming of the parameters I don't think the API actually works as you described:
ToChildBlockJoinQuery(Query parentQuery, Filter parentsFilter, boolean doScores)
If the filter was intended to filter the child docs I think it would be called childFilter no?
I think the use of the CachingWrappingFilter in the example I got from Mike McCandless' blog post was the real cause of the exception I was seeing (maybe things have changed internally since that post). I finally noticed a mention of the FixedBitSetCachingWrapperFilter in the description of the ToChildBlockJoinQuery constructor in the API docs. When I changed to using a filter produced by the FixedBitSetCachingWrapperFilter class the IllegalStateException no longer occurs and I get the child docs using ToChildBlockJoinQuery with a parent doc filter and parent doc query and results look correctly limited by the parent constraints. For example:
...
Gub-Gub's Book: An Encyclopedia of Food (Fictional work), Fictional work, 119320101
by: Lofting, Hugh - NP, American, Writer
The Story of Doctor Dolittle, Being the History of His Peculiar Life at Home and Astonishing Adventures in Foreign Parts (Novel), Novel, 119200101
by: Lofting, Hugh - NP, American, Writer
The Voyages of Doctor Dolittle (Novel), Novel, 119220101
by: Lofting, Hugh - NP, American, Writer
The Story of Doctor Dolittle (Novel), Novel, 119200101
by: Lofting, Hugh - NP, American, Writer
...
Mister Beers (Poem), Poem, null
by: Lofting, Hugh - NP, American, Writer
The Twilight of Magic (Novel), Novel, 119300101
by: Lofting, Hugh - NP, American, Writer
Picnic (Lofting, Hugh) (Poem), Poem, null
by: Lofting, Hugh - NP, American, Writer
The Impossible Patriotism Project (Picture story), Picture story, 120070101
A Skeleton in God's Closet: A Novel (Novel), Novel, 119940101
by: Maier, Paul Luther - NP, American, null
Pontius Pilate (Novel), Novel, 119680101
by: Maier, Paul Luther - NP, American, null
...
Josephus: The Essential Writings (Collection), Collection, 119880101
by: Maier, Paul Luther - NP, American, null
She Said the Geese (Poem), Poem, null
by: Lifshin, Lyn - NP, American, Poet
She Said She Could See Music (Poem), Poem, null
by: Lifshin, Lyn - NP, American, Poet
...
However I see no way to further limit the children as you describe. If I use "a query that matches the set of parents and a filter that matches the set of children" as you suggest I get no results back. I think your description of how it should work makes complete sense, but that is not what I'm seeing when I try it. Here's the code that produced the above output:
private void runToChildBlockJoinQuery(String indexPath) throws IOException {
FSDirectory dir = FSDirectory.open(new File(indexPath));
IndexReader reader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(reader);
TermQuery parentFilterQuery = new TermQuery(new Term("AGTY", "np"));
BooleanQuery parentQuery = new BooleanQuery();
parentQuery.add(new TermQuery(new Term("AGTY", "np")), Occur.MUST);
parentQuery.add(new TermQuery(new Term("NT", "american")), Occur.MUST);
Filter parentFilter = new FixedBitSetCachingWrapperFilter(new QueryWrapperFilter(parentFilterQuery));
ToChildBlockJoinQuery tcbjq = new ToChildBlockJoinQuery(parentQuery, parentFilter, true);
TopDocs worksDocs = searcher.search(tcbjq, 5000);
System.out.println("\n*ToChildBlockJoinQuery hit count = " + worksDocs.scoreDocs.length);
displayWorks(reader, searcher, worksDocs);
}
private void displayWorks(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
String tw = reader.document(worksDocs.scoreDocs[i].doc).get("TW");
String pd = reader.document(worksDocs.scoreDocs[i].doc).get("PD");
String crid = reader.document(worksDocs.scoreDocs[i].doc).get("CRID");
TopDocs creatorDocs = searcher.search(new TermQuery(new Term("ABID", crid)), Integer.MAX_VALUE);
System.out.println("\n" + agdn + ", " + tw + ", " + pd);
displayCreators(reader, searcher, creatorDocs);
}
}
private void displayCreators(IndexReader reader, IndexSearcher searcher, TopDocs worksDocs) throws IOException {
for (int i = 0; i < worksDocs.scoreDocs.length; i++) {
String agdn = reader.document(worksDocs.scoreDocs[i].doc).get("AGDN");
String agty = reader.document(worksDocs.scoreDocs[i].doc).get("AGTY");
String nt = reader.document(worksDocs.scoreDocs[i].doc).get("NT");
String poc = reader.document(worksDocs.scoreDocs[i].doc).get("POC");
System.out.println("\tby: " + agdn + " - " + agty + ", " +nt + ", " + poc);
}
}
When I try to use ToParentBlockJoinQuery I don't get any results either and it is not what I really want anyway, I want the child documents limited by the parent documents.
ToChildBlockJoinQuery almost gives me what I want, but I really need to be able to filter the child docs returned as well as the parent from which they came. If you (or anybody) still thinks I'm doing it wrong please let me know. If I should file a bug report also let me know that, I have a small index I can provide if it is useful. Thanks again for your help.
Jim
________________________________________
From: Gregory Dearing [gregdearing@gmail.com]
Sent: Wednesday, January 21, 2015 6:59 PM
To: java-user@lucene.apache.org
Subject: Re: ToChildBlockJoinQuery question
Jim,
I think you hit the nail on the head... that's not what BlockJoinQueries do.
If you're wanting to search for children and join to their parents... then
use ToParentBlockJoinQuery, with a query that matches the set of children
and a filter that matches the set of parents.
If you're searching for parents, then joining to their children... then use
ToChildBlockJoinQuery, with a query that matches the set of parents and a
filter that matches the set of children.
When you add related documents to the index (via addDocuments), make that
children are added before their parents.
The reason all the above is necessary is that it makes it possible to have
a nested hierarchy of relationships (ie. Parents have Children, which have
Children of their own). You need a query to indicate which part of the
hierarchy you're starting from, and a filter indicating which part of the
hierarchy you're joining to.
Also, you will always get an exception if your query and your filter both
match the same document. A child can't be its own parent.
BlockJoin is a very powerful feature, but what it's really doing is
modelling relationships using an index that doesn't know what a
relationship is. The relationships are determined by a combination of the
order that you indexed the block, and the format of your query. This
disjoin can lead to some weird behavior if you're not absolutely sure how
it works.
Thanks,
Greg
On Wed, Jan 21, 2015 at 4:34 PM, McKinley, James T <
james.mckinley@cengage.com> wrote:
>
> Am I understanding how this is supposed to work? What I think I am (and
> should be) doing is providing a query and filter that specifies the parent
> docs and the ToChildBlockJoinQuery should return me all the child docs for
> the resulting parent docs. Is this correct? The reason I think I'm not
> understanding is that I don't see why I need both a filter and a query to
> specify the parent docs when a single query or filter should suffice. Am I
> misunderstanding what parentQuery and parentFilter mean, they both refer to
> parent docs right?
>
> Jim
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ToChildBlockJoinQuery question
Posted by Gregory Dearing <gr...@gmail.com>.
Hey Mike,
My fault... I wasn't paying attention and thought I was replying to a
response from James. No wonder it reminded me of our last conversation. :)
-Greg
On Thu, Jan 22, 2015 at 10:37 AM, Michael Sokolov <
msokolov@safaribooksonline.com> wrote:
> Yeah I know -- we've been around this block before. I agree that the
> whole block indexing/searching feature is a bit confusing, trappy and
> error-prone, and it may be helpful to have those boundary conditions as
> signposts, but in my case relaxing the restriction enabled me to execute
> the queries I want without having to write a lot of awkward extensions to
> my indexing code. That code uses Python's haystack, which is based on
> django models, and in order to comply with the parent-not-its-child idea, I
> would have had to introduce dummy documents to stand in as the parents,
> something that isn't at all natural or straightforward in that
> django/haystack view of the world. Maybe the enforcement of that
> restriction could be relaxed according to an option in the query
> constructor.
>
> -Mike
>
Re: ToChildBlockJoinQuery question
Posted by Michael Sokolov <ms...@safaribooksonline.com>.
Yeah I know -- we've been around this block before. I agree that the
whole block indexing/searching feature is a bit confusing, trappy and
error-prone, and it may be helpful to have those boundary conditions as
signposts, but in my case relaxing the restriction enabled me to execute
the queries I want without having to write a lot of awkward extensions
to my indexing code. That code uses Python's haystack, which is based
on django models, and in order to comply with the parent-not-its-child
idea, I would have had to introduce dummy documents to stand in as the
parents, something that isn't at all natural or straightforward in that
django/haystack view of the world. Maybe the enforcement of that
restriction could be relaxed according to an option in the query
constructor.
-Mike
On 1/22/15 10:29 AM, Gregory Dearing wrote:
> Mike,
>
> I agree that it's not absolutely necessary to enforce children not being
> their own parent. I was just trying to describe the current
> implementation, and why you were seeing exceptions.
>
> The difference is mostly philosophical. The advantage of the current
> approach (in my opinion) is that it the BlockJoin mechanic has a lot of
> terrible edge cases if used naively, and enforcing "child can't be its own
> parent" can help catch quite a few of them.
>
> I had a discussion on this list on the same topic, which might be useful: Re:
> including self-joins in parent/child queries
> <http://mail-archives.apache.org/mod_mbox/lucene-java-user/201412.mbox/%3CCAASL1-_ppmCNQ3aPJjFbT3AdB4pgaSpvE-8o5R9gV5kLDpf++A@mail.gmail.com%3E>
>
> -Greg
>
> On Wed, Jan 21, 2015 at 7:59 PM, Michael Sokolov <
> msokolov@safaribooksonline.com> wrote:
>
>> On 1/21/2015 6:59 PM, Gregory Dearing wrote:
>>
>>> Jim,
>>>
>>> I think you hit the nail on the head... that's not what BlockJoinQueries
>>> do.
>>>
>>> If you're wanting to search for children and join to their parents... then
>>> use ToParentBlockJoinQuery, with a query that matches the set of children
>>> and a filter that matches the set of parents.
>>>
>>> If you're searching for parents, then joining to their children... then
>>> use
>>> ToChildBlockJoinQuery, with a query that matches the set of parents and a
>>> filter that matches the set of children.
>>>
>>> When you add related documents to the index (via addDocuments), make that
>>> children are added before their parents.
>>>
>>> The reason all the above is necessary is that it makes it possible to have
>>> a nested hierarchy of relationships (ie. Parents have Children, which have
>>> Children of their own). You need a query to indicate which part of the
>>> hierarchy you're starting from, and a filter indicating which part of the
>>> hierarchy you're joining to.
>>>
>>> Also, you will always get an exception if your query and your filter both
>>> match the same document. A child can't be its own parent.
>>>
>> That's true for the existing implementation, but seems unnecessary from
>> what I can tell. See https://github.com/safarijv/
>> ifpress-solr-plugin/blob/master/src/main/java/com/
>> ifactory/press/db/solr/search/SafariBlockJoinQuery.java for a variant
>> that allows a child to be its own parent.
>>
>> -Mike
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ToChildBlockJoinQuery question
Posted by Gregory Dearing <gr...@gmail.com>.
Mike,
I agree that it's not absolutely necessary to enforce children not being
their own parent. I was just trying to describe the current
implementation, and why you were seeing exceptions.
The difference is mostly philosophical. The advantage of the current
approach (in my opinion) is that it the BlockJoin mechanic has a lot of
terrible edge cases if used naively, and enforcing "child can't be its own
parent" can help catch quite a few of them.
I had a discussion on this list on the same topic, which might be useful: Re:
including self-joins in parent/child queries
<http://mail-archives.apache.org/mod_mbox/lucene-java-user/201412.mbox/%3CCAASL1-_ppmCNQ3aPJjFbT3AdB4pgaSpvE-8o5R9gV5kLDpf++A@mail.gmail.com%3E>
-Greg
On Wed, Jan 21, 2015 at 7:59 PM, Michael Sokolov <
msokolov@safaribooksonline.com> wrote:
> On 1/21/2015 6:59 PM, Gregory Dearing wrote:
>
>> Jim,
>>
>> I think you hit the nail on the head... that's not what BlockJoinQueries
>> do.
>>
>> If you're wanting to search for children and join to their parents... then
>> use ToParentBlockJoinQuery, with a query that matches the set of children
>> and a filter that matches the set of parents.
>>
>> If you're searching for parents, then joining to their children... then
>> use
>> ToChildBlockJoinQuery, with a query that matches the set of parents and a
>> filter that matches the set of children.
>>
>> When you add related documents to the index (via addDocuments), make that
>> children are added before their parents.
>>
>> The reason all the above is necessary is that it makes it possible to have
>> a nested hierarchy of relationships (ie. Parents have Children, which have
>> Children of their own). You need a query to indicate which part of the
>> hierarchy you're starting from, and a filter indicating which part of the
>> hierarchy you're joining to.
>>
>> Also, you will always get an exception if your query and your filter both
>> match the same document. A child can't be its own parent.
>>
> That's true for the existing implementation, but seems unnecessary from
> what I can tell. See https://github.com/safarijv/
> ifpress-solr-plugin/blob/master/src/main/java/com/
> ifactory/press/db/solr/search/SafariBlockJoinQuery.java for a variant
> that allows a child to be its own parent.
>
> -Mike
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: ToChildBlockJoinQuery question
Posted by Michael Sokolov <ms...@safaribooksonline.com>.
On 1/21/2015 6:59 PM, Gregory Dearing wrote:
> Jim,
>
> I think you hit the nail on the head... that's not what BlockJoinQueries do.
>
> If you're wanting to search for children and join to their parents... then
> use ToParentBlockJoinQuery, with a query that matches the set of children
> and a filter that matches the set of parents.
>
> If you're searching for parents, then joining to their children... then use
> ToChildBlockJoinQuery, with a query that matches the set of parents and a
> filter that matches the set of children.
>
> When you add related documents to the index (via addDocuments), make that
> children are added before their parents.
>
> The reason all the above is necessary is that it makes it possible to have
> a nested hierarchy of relationships (ie. Parents have Children, which have
> Children of their own). You need a query to indicate which part of the
> hierarchy you're starting from, and a filter indicating which part of the
> hierarchy you're joining to.
>
> Also, you will always get an exception if your query and your filter both
> match the same document. A child can't be its own parent.
That's true for the existing implementation, but seems unnecessary from
what I can tell. See
https://github.com/safarijv/ifpress-solr-plugin/blob/master/src/main/java/com/ifactory/press/db/solr/search/SafariBlockJoinQuery.java
for a variant that allows a child to be its own parent.
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ToChildBlockJoinQuery question
Posted by Gregory Dearing <gr...@gmail.com>.
Jim,
I think you hit the nail on the head... that's not what BlockJoinQueries do.
If you're wanting to search for children and join to their parents... then
use ToParentBlockJoinQuery, with a query that matches the set of children
and a filter that matches the set of parents.
If you're searching for parents, then joining to their children... then use
ToChildBlockJoinQuery, with a query that matches the set of parents and a
filter that matches the set of children.
When you add related documents to the index (via addDocuments), make that
children are added before their parents.
The reason all the above is necessary is that it makes it possible to have
a nested hierarchy of relationships (ie. Parents have Children, which have
Children of their own). You need a query to indicate which part of the
hierarchy you're starting from, and a filter indicating which part of the
hierarchy you're joining to.
Also, you will always get an exception if your query and your filter both
match the same document. A child can't be its own parent.
BlockJoin is a very powerful feature, but what it's really doing is
modelling relationships using an index that doesn't know what a
relationship is. The relationships are determined by a combination of the
order that you indexed the block, and the format of your query. This
disjoin can lead to some weird behavior if you're not absolutely sure how
it works.
Thanks,
Greg
On Wed, Jan 21, 2015 at 4:34 PM, McKinley, James T <
james.mckinley@cengage.com> wrote:
>
> Am I understanding how this is supposed to work? What I think I am (and
> should be) doing is providing a query and filter that specifies the parent
> docs and the ToChildBlockJoinQuery should return me all the child docs for
> the resulting parent docs. Is this correct? The reason I think I'm not
> understanding is that I don't see why I need both a filter and a query to
> specify the parent docs when a single query or filter should suffice. Am I
> misunderstanding what parentQuery and parentFilter mean, they both refer to
> parent docs right?
>
> Jim
>
RE: ToChildBlockJoinQuery question
Posted by "McKinley, James T" <ja...@cengage.com>.
Hi Greg,
Thanks for responding to my question. I added some extra conditions to the IndexRunnable run method, namely I required AGTY:np in the source query for the parent docs and required that both the creatorDocs and workDocs actually contain documents or else the addDocuments call would never be made:
public void run() {
IndexSearcher searcher = new IndexSearcher(reader);
try {
int count = 0;
for (String crid : crids) {
List<Document> docs = new ArrayList<>();
BooleanQuery abidQuery = new BooleanQuery();
abidQuery.add(new TermQuery(new Term("ABID", crid)), Occur.MUST);
abidQuery.add(new TermQuery(new Term("AGPR", "true")), Occur.MUST);
abidQuery.add(new TermQuery(new Term("AGTY", "np")), Occur.MUST);
TermQuery cridQuery = new TermQuery(new Term("CRID", crid));
TopDocs creatorDocs = searcher.search(abidQuery, Integer.MAX_VALUE);
TopDocs workDocs = searcher.search(cridQuery, Integer.MAX_VALUE);
if ((creatorDocs.scoreDocs.length > 0) && (workDocs.scoreDocs.length > 0)) {
for (int i = 0; i < workDocs.scoreDocs.length; i++) {
docs.add(reader.document(workDocs.scoreDocs[i].doc));
}
docs.add(reader.document(creatorDocs.scoreDocs[0].doc));
writer.addDocuments(docs);
if (++count % 100 == 0) {
System.out.println(id + " = " + count);
writer.commit();
}
}
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
I then modified the runToChildBlockJoinQuery method to first perform a search with the parent query and parent filter. Then using the id of each parent named person document I did a query for the named works with that creator id (essentially reversing the query that was done to create the BlockJoin index) and I do indeed get works back for every named person that passes the parent query and filter. However I still get the IllegalStateException complaining about a non-FixedBitSet doc id set when doing the ToChildBlockJoinQuery. Here is that code:
private void runToChildBlockJoinQuery(String indexPath) throws IOException {
FSDirectory dir = FSDirectory.open(new File(indexPath));
IndexReader reader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(reader);
TermQuery parentFilterQuery = new TermQuery(new Term("AGTY", "np"));
TermQuery parentQuery = new TermQuery(new Term("NT", "american"));
Filter parentFilter = new CachingWrapperFilter(new QueryWrapperFilter(parentFilterQuery));
TopDocs creatorDocs = searcher.search(parentQuery, parentFilter, Integer.MAX_VALUE);
for (ScoreDoc scoreDoc : creatorDocs.scoreDocs) {
String[] ids = reader.document(scoreDoc.doc).getValues("ABID");
BooleanQuery cridQuery = new BooleanQuery();
for (String id : ids) {
cridQuery.add(new TermQuery(new Term("CRID", id)), Occur.SHOULD);
}
TopDocs worksDocs = searcher.search(cridQuery, Integer.MAX_VALUE);
System.out.println(worksDocs.scoreDocs.length);
}
ToChildBlockJoinQuery tcbjq = new ToChildBlockJoinQuery(parentQuery, parentFilter, true);
TopDocs worksDocs = searcher.search(tcbjq, Integer.MAX_VALUE); // ==> IllegalStateException
}
So I think all the parent docs have child docs and they should have been indexed in the same addDocuments call with the parent being the last doc in the list. Then, on a lark, I just made the parentFilterQuery and the parentQuery the same and still got the exception.
Am I understanding how this is supposed to work? What I think I am (and should be) doing is providing a query and filter that specifies the parent docs and the ToChildBlockJoinQuery should return me all the child docs for the resulting parent docs. Is this correct? The reason I think I'm not understanding is that I don't see why I need both a filter and a query to specify the parent docs when a single query or filter should suffice. Am I misunderstanding what parentQuery and parentFilter mean, they both refer to parent docs right?
I attempted to attach a small tar.gz file (< 1MB) to this message that contained a 100 parent index (~10,000 docs total) that gives the exception with my block join query, but the mailing list rejected my message, if there's a better place to send/upload this index let me know and I surely will. Thanks again for any help.
Jim
________________________________________
From: Gregory Dearing [gregdearing@gmail.com]
Sent: Wednesday, January 21, 2015 1:01 PM
To: java-user@lucene.apache.org
Subject: Re: ToChildBlockJoinQuery question
James,
I haven't actually ran your example, but I think the source problem is that
your source query ("NT:American") is hitting documents that have no
children.
The reason the exception is so weird is that one of your index segments
contains zero documents that match your filter. Specifically, there's an
index segment containing docs matching "NT:american", but with no documents
matching "AGTY:np".
This will cause CachingWrapperFilter, which normally returns a FixedBitSet,
to instead return a generic "Empty" DocIdSet. Which leads to the exception
from ToChildBlockJoinQuery.
The summary is, make sure that your source query only hits documents that
were actually added using 'addDocuments()'. Since it looks like you're
extracting your block relationships from the existing index, that might
mean that you'll need to add some extra metadata to the newly created docs
instead of just cloning what already exists.
-Greg
On Wed, Jan 21, 2015 at 10:00 AM, McKinley, James T <
james.mckinley@cengage.com> wrote:
> Hi,
>
> I'm attempting to use ToChildBlockJoinQuery in Lucene 4.8.1 by following
> Mike McCandless' blog post:
>
>
> http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html
>
> I have a set of child documents which are named works and a set of parent
> documents which are named persons that are the creators of the named
> works. The parent document has a nationality and the child document does
> not. I want to query the children (named works) limiting by the
> nationality of the parent (named person). I've indexed the documents as
> follows (I'm pulling the docs from an existing index):
>
> private void createNamedWorkIndex(String srcIndexPath, String
> destIndexPath) throws IOException {
> FSDirectory srcDir = FSDirectory.open(new
> File(srcIndexPath));
> FSDirectory destDir = FSDirectory.open(new
> File(destIndexPath));
>
> IndexReader reader = DirectoryReader.open(srcDir);
>
> Version version = Version.LUCENE_48;
> IndexWriterConfig conf = new IndexWriterConfig(version,
> new StandardTextAnalyzer(version));
>
> Set<String> crids = getCreatorIds(reader);
>
> String[] crida = crids.toArray(new String[crids.size()]);
>
> int numThreads = 24;
> ExecutorService executor =
> Executors.newFixedThreadPool(numThreads);
>
> int numCrids = crids.size();
> int batchSize = numCrids / numThreads;
> int remainder = numCrids % numThreads;
>
> System.out.println("Inserting work/creator blocks using "
> + numThreads + " threads...");
> try (IndexWriter writer = new IndexWriter(destDir, conf)){
> for (int i = 0; i < numThreads; i++) {
> String[] cridRange;
> if (i == numThreads - 1) {
> cridRange =
> Arrays.copyOfRange(crida, i*batchSize, ((i+1)*batchSize - 1) + remainder);
> } else {
> cridRange =
> Arrays.copyOfRange(crida, i*batchSize, ((i+1)*batchSize - 1));
> }
> String id = "" + ((char)('A' + i));
> Runnable indexer = new IndexRunnable(id ,
> reader, writer, new HashSet<String>(Arrays.asList(cridRange)));
> executor.execute(indexer);
> }
> executor.shutdown();
> executor.awaitTermination(2, TimeUnit.HOURS);
> } catch (Exception e) {
> executor.shutdownNow();
> throw new RuntimeException(e);
> } finally {
> reader.close();
> srcDir.close();
> destDir.close();
> }
>
> System.out.println("Done!");
> }
>
> public static class IndexRunnable implements Runnable {
> private String id;
> private IndexReader reader;
> private IndexWriter writer;
> private Set<String> crids;
>
> public IndexRunnable(String id, IndexReader reader,
> IndexWriter writer, Set<String> crids) {
> this.id = id;
> this.reader = reader;
> this.writer = writer;
> this.crids = crids;
> }
>
> @Override
> public void run() {
> IndexSearcher searcher = new IndexSearcher(reader);
>
> try {
> int count = 0;
> for (String crid : crids) {
> List<Document> docs = new
> ArrayList<>();
>
> BooleanQuery abidQuery = new
> BooleanQuery();
> abidQuery.add(new TermQuery(new
> Term("ABID", crid)), Occur.MUST);
> abidQuery.add(new TermQuery(new
> Term("AGPR", "true")), Occur.MUST);
>
> TermQuery cridQuery = new
> TermQuery(new Term("CRID", crid));
>
> TopDocs creatorDocs =
> searcher.search(abidQuery, Integer.MAX_VALUE);
> TopDocs workDocs =
> searcher.search(cridQuery, Integer.MAX_VALUE);
>
> for (int i = 0; i <
> workDocs.scoreDocs.length; i++) {
>
> docs.add(reader.document(workDocs.scoreDocs[i].doc));
> }
>
> if (creatorDocs.scoreDocs.length >
> 0) {
>
> docs.add(reader.document(creatorDocs.scoreDocs[0].doc));
> }
>
> writer.addDocuments(docs);
> if (++count % 100 == 0) {
> System.out.println(id + "
> = " + count);
> writer.commit();
> }
> }
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> I then attempt to perform a block join query as follows:
>
> private void runToChildBlockJoinQuery(String indexPath) throws
> IOException {
> FSDirectory dir = FSDirectory.open(new File(indexPath));
> IndexReader reader = DirectoryReader.open(dir);
> IndexSearcher searcher = new IndexSearcher(reader);
>
> TermQuery parentQuery = new TermQuery(new Term("NT",
> "american"));
> TermQuery parentFilterQuery = new TermQuery(new
> Term("AGTY", "np"));
> Filter parentFilter = new CachingWrapperFilter(new
> QueryWrapperFilter(parentFilterQuery));
>
> ToChildBlockJoinQuery tcbjq = new
> ToChildBlockJoinQuery(parentQuery, parentFilter, true);
>
> TopDocs worksDocs = searcher.search(tcbjq, 20);
>
> displayWorks(reader, searcher, worksDocs);
> }
>
> and I get the following exception:
>
> Exception in thread "main" java.lang.IllegalStateException: parentFilter
> must return FixedBitSet; got org.apache.lucene.util.WAH8DocIdSet@34e671de
> at
> org.apache.lucene.search.join.ToChildBlockJoinQuery$ToChildBlockJoinWeight.scorer(ToChildBlockJoinQuery.java:148)
> at org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> at
> BlockJoinQueryTester.runToChildBlockJoinQuery(BlockJoinQueryTester.java:73)
> at BlockJoinQueryTester.main(BlockJoinQueryTester.java:40)
>
> I don't understand what I'm doing wrong and what a "FixedBitSet" is and
> why I don't get one out of my filter. Is FixedBitSet a special kind of
> OpenBitSet and what does "fixed" mean in this context? Thanks for any help.
>
> Jim
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ToChildBlockJoinQuery question
Posted by Gregory Dearing <gr...@gmail.com>.
James,
I haven't actually ran your example, but I think the source problem is that
your source query ("NT:American") is hitting documents that have no
children.
The reason the exception is so weird is that one of your index segments
contains zero documents that match your filter. Specifically, there's an
index segment containing docs matching "NT:american", but with no documents
matching "AGTY:np".
This will cause CachingWrapperFilter, which normally returns a FixedBitSet,
to instead return a generic "Empty" DocIdSet. Which leads to the exception
from ToChildBlockJoinQuery.
The summary is, make sure that your source query only hits documents that
were actually added using 'addDocuments()'. Since it looks like you're
extracting your block relationships from the existing index, that might
mean that you'll need to add some extra metadata to the newly created docs
instead of just cloning what already exists.
-Greg
On Wed, Jan 21, 2015 at 10:00 AM, McKinley, James T <
james.mckinley@cengage.com> wrote:
> Hi,
>
> I'm attempting to use ToChildBlockJoinQuery in Lucene 4.8.1 by following
> Mike McCandless' blog post:
>
>
> http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html
>
> I have a set of child documents which are named works and a set of parent
> documents which are named persons that are the creators of the named
> works. The parent document has a nationality and the child document does
> not. I want to query the children (named works) limiting by the
> nationality of the parent (named person). I've indexed the documents as
> follows (I'm pulling the docs from an existing index):
>
> private void createNamedWorkIndex(String srcIndexPath, String
> destIndexPath) throws IOException {
> FSDirectory srcDir = FSDirectory.open(new
> File(srcIndexPath));
> FSDirectory destDir = FSDirectory.open(new
> File(destIndexPath));
>
> IndexReader reader = DirectoryReader.open(srcDir);
>
> Version version = Version.LUCENE_48;
> IndexWriterConfig conf = new IndexWriterConfig(version,
> new StandardTextAnalyzer(version));
>
> Set<String> crids = getCreatorIds(reader);
>
> String[] crida = crids.toArray(new String[crids.size()]);
>
> int numThreads = 24;
> ExecutorService executor =
> Executors.newFixedThreadPool(numThreads);
>
> int numCrids = crids.size();
> int batchSize = numCrids / numThreads;
> int remainder = numCrids % numThreads;
>
> System.out.println("Inserting work/creator blocks using "
> + numThreads + " threads...");
> try (IndexWriter writer = new IndexWriter(destDir, conf)){
> for (int i = 0; i < numThreads; i++) {
> String[] cridRange;
> if (i == numThreads - 1) {
> cridRange =
> Arrays.copyOfRange(crida, i*batchSize, ((i+1)*batchSize - 1) + remainder);
> } else {
> cridRange =
> Arrays.copyOfRange(crida, i*batchSize, ((i+1)*batchSize - 1));
> }
> String id = "" + ((char)('A' + i));
> Runnable indexer = new IndexRunnable(id ,
> reader, writer, new HashSet<String>(Arrays.asList(cridRange)));
> executor.execute(indexer);
> }
> executor.shutdown();
> executor.awaitTermination(2, TimeUnit.HOURS);
> } catch (Exception e) {
> executor.shutdownNow();
> throw new RuntimeException(e);
> } finally {
> reader.close();
> srcDir.close();
> destDir.close();
> }
>
> System.out.println("Done!");
> }
>
> public static class IndexRunnable implements Runnable {
> private String id;
> private IndexReader reader;
> private IndexWriter writer;
> private Set<String> crids;
>
> public IndexRunnable(String id, IndexReader reader,
> IndexWriter writer, Set<String> crids) {
> this.id = id;
> this.reader = reader;
> this.writer = writer;
> this.crids = crids;
> }
>
> @Override
> public void run() {
> IndexSearcher searcher = new IndexSearcher(reader);
>
> try {
> int count = 0;
> for (String crid : crids) {
> List<Document> docs = new
> ArrayList<>();
>
> BooleanQuery abidQuery = new
> BooleanQuery();
> abidQuery.add(new TermQuery(new
> Term("ABID", crid)), Occur.MUST);
> abidQuery.add(new TermQuery(new
> Term("AGPR", "true")), Occur.MUST);
>
> TermQuery cridQuery = new
> TermQuery(new Term("CRID", crid));
>
> TopDocs creatorDocs =
> searcher.search(abidQuery, Integer.MAX_VALUE);
> TopDocs workDocs =
> searcher.search(cridQuery, Integer.MAX_VALUE);
>
> for (int i = 0; i <
> workDocs.scoreDocs.length; i++) {
>
> docs.add(reader.document(workDocs.scoreDocs[i].doc));
> }
>
> if (creatorDocs.scoreDocs.length >
> 0) {
>
> docs.add(reader.document(creatorDocs.scoreDocs[0].doc));
> }
>
> writer.addDocuments(docs);
> if (++count % 100 == 0) {
> System.out.println(id + "
> = " + count);
> writer.commit();
> }
> }
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> I then attempt to perform a block join query as follows:
>
> private void runToChildBlockJoinQuery(String indexPath) throws
> IOException {
> FSDirectory dir = FSDirectory.open(new File(indexPath));
> IndexReader reader = DirectoryReader.open(dir);
> IndexSearcher searcher = new IndexSearcher(reader);
>
> TermQuery parentQuery = new TermQuery(new Term("NT",
> "american"));
> TermQuery parentFilterQuery = new TermQuery(new
> Term("AGTY", "np"));
> Filter parentFilter = new CachingWrapperFilter(new
> QueryWrapperFilter(parentFilterQuery));
>
> ToChildBlockJoinQuery tcbjq = new
> ToChildBlockJoinQuery(parentQuery, parentFilter, true);
>
> TopDocs worksDocs = searcher.search(tcbjq, 20);
>
> displayWorks(reader, searcher, worksDocs);
> }
>
> and I get the following exception:
>
> Exception in thread "main" java.lang.IllegalStateException: parentFilter
> must return FixedBitSet; got org.apache.lucene.util.WAH8DocIdSet@34e671de
> at
> org.apache.lucene.search.join.ToChildBlockJoinQuery$ToChildBlockJoinWeight.scorer(ToChildBlockJoinQuery.java:148)
> at org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> at
> BlockJoinQueryTester.runToChildBlockJoinQuery(BlockJoinQueryTester.java:73)
> at BlockJoinQueryTester.main(BlockJoinQueryTester.java:40)
>
> I don't understand what I'm doing wrong and what a "FixedBitSet" is and
> why I don't get one out of my filter. Is FixedBitSet a special kind of
> OpenBitSet and what does "fixed" mean in this context? Thanks for any help.
>
> Jim
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>